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As soci^ty^s dependence on tecnnology increases, the. need for competent 
technicians vho can maintain and rapair complex systems increases as well. 
Present ineth^ds of teaching trcxibleshooting/debuigging remain primitive and 
expensive, relying on students to discover effective and efficient problem- 
solving methods by observation and practice %n relatively unstructured ^ 
^vironisents* The goal of the present project was to idestify the types of 
knowledge, necessaiy and xiseful for ccmpetent troubleshooting/debugging and 
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tc eJtSainfi _ r.ev apprcacfaes tc fomal instr^otior. rd-gnt infl-jence trxe attaih 
aent ;.f cocpetence b^,' sr^dertsy In particular, researcr. f c-cosed. ;n; tns 
rclfe zt general strategies in/trr^leshccting/det'oggiaig and hr- tney iiigbt 'ce 
represented ar.d ta-j^nt explicitly ar.d directly in crder tc atoid' tne cost aad- 
ctnsr^dravtacKs cf leamLr^ indirectly ty cbser-.-aticn. ^nl^practioe. 

?*lated y;r£ cn trcucleshc-cting/deb-ogging vas exacined and in conjunctisr^' 
-vitn a .'.gical aralysis ccntrir-ted tc a cnaracterization c'f . trcrableshoctin^' ' 
deb-^gglr^ prcblenr and proDle^-scl-^-.g prccesses tnat enpnasi^es tneir j?ener- 
ality acrc-s a c-E'c^r cf tecnnical fields ar^ infernal ccn^exts. Tne analyses 
&l;c- s^e-t; -nat dgc'-g^ir^ "is ^ fundamental aspect cf aiinost all ..leamLng 
V.c prcclez £::.-.-lng. Zr.^ res-^t cf tre analysis vas tr£ fcjrcr-l^tdcn cf an" 
infc-iaticr.-prc-cessir^ =cael cf a general trc"ableshcc-ting/deb-£ging strategy, ' 
vnicn describ-es trs tv-p^s cf ^^ascnL-^ pp-cesse's nee-ded, sccie 5f ti^ factcrs 
gc-.-emit* selecticn cf a.terr.ati-.-e crcce'ses in sclv-lne a i:rcble=, and an 
exp:^cijb centre- ^trateg;,-. •• " " / 

Ijttensi-.'e ezaiiinaticn it_ a ccr?--L3 cr: data frcs: srodents learning carx^^ter 
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^. - --nderta/.en, ar.d ^cce f-rtner lirJ.t«c der-^Lng data C-er^ ccl- 
-ectec free rctn .exp-erienced inexperienced prc^grat-srs . Tnese data are 
ccncist^nt -"itr-ia^ nv^ctr^s is tnat expert cer^eips dc net necessarily na-.-e 
s -purler general strategies, ct.t instead tr^t t.-jeir extort is-e derives free - 
specific ar.i scc^tii^s idies:.r.cratic .'j-.e---ledge acq-^red tr^o^jn 9?:perienee, 
^rsxp^riencsc;. pregra=ners lac/. t:^s imcvledge, r.it in additien scc^ cf trja:;,.^ 
nav-e.a eefecttve general strateg;,- &s veil, "in ar. atte=t^. tc cbtair. "a rlacrcras 
c.'.aracterizatieb. cf tne iifferences and defects in tne der^inj strategy cf 
t.n^r pre £ra=r.-ng sti^ents, ar. '^ffert vas race tc apply a zccel-crient«d data 
analysis n^t.'vcc repcrted in t.-ss literar^re. Ecve-.-er, tbs method vas --r^uc- 
cescf.l fcr tne data available and eay nave zcre basic liciiteticns . As a 
censec-..ence enly -iif emal' cenclusicns ab=c"..t tns defec^i-.-e strategies used r,- ' 
se--e inexp«rier-ced deV^ere cctild "ce dfe-.'elepe.dj 'It they are deficient in ' 
pregrae tester^ ar.d se fail tc fiied b^s ; '2] tr^y do net collect cr -ose a-/ail-j 
ac^e-cata acr-t the effects cf a "f-g tc constrain their reasoning; '3} -Jiey 

— ^ attempting minor a:ii societiises irrational repairs; 
~' tr^y ce not . bac^trac/. veil froc unsuccessfxil repair it^e=pts. 
A szall-scale Ef^y vas cond;..cted to determir.«e the" effects of crese^^tlng 
a tutorial text, vnicn trxp.rcit ly descri-Des a fev. general heuristics destgned 
to correct ttese strateg;.- deficits, to no-rice progracr^rs . Tr^ dare indicate 
a marginal increase in the apparent -j^e of sees of tr^ neuristics by toe nrc- 
grarisers vne studied tne text cecpared to a gro-ap v^o did no*. In addition 



lents v-^ro ger^rally ?avcrable zc presenting 



prct.^-stlv-ir.g strategies explicitly, as tt^y «ere ir. t£^ r\*tcrial. Hcv^-.-er, 
tr^ success cf tne grc--.ps in sclv-ing deb'ugging test pitJblezs aid'^Got differ. . 
Tr^re v-ere se-.-eral -ef^.cdcicgical liiiitaticnB and prcb'leiLS encountered in tbe 
st'^y vrJ.cn f^^rtr^r ccnfT^d tne results. More ger^ral s^thodolc^ical issues 
fcr 3fadies designed tc investigate instruction in troubleshcctini/aebuggir^ 
alsc b^caice apparent, Cr.e cf tr;e ni-ost irrpcrtact is analysis cf c9Giplex 
prcblerj-sc^v-ir^ data: if tne causes cf differences Ln ability are tc be spec- 
ified in detail ar.d if_^De effects cf direct prcbie-lsGl"^ir£ instruction ar^ 
tc be assessed, then it vill be necessary tc perfect mcd^l^bas^d data analysis 
n^etncds. 
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As society's dependence op technology Increases, the need for 
cocpetent teciinlcians who can nalntain and repair cosplex syst^eas 
lncre3ses_ as wel^. Present methods of i^a'chlng 

troubresnooting/debugginR renalo prtoitlve and expensive, relying on 
sttidents to discover effective and efficient problem-solving netbods by 
observation and practlc^in relatively usstruc|;ured environments. Tm - 
^^goai of the present project */a8 to Identify the types of kodwledge 
necessary and useful for conpetent t roubleshooting/debugging ^d to ^ 
exanine how new approaches to forrial instruction sight in&luenpe the 
attaincrent of conpetence by students. In particular, the research 
focu^d on tne role of general strategies in troubleshooting/debugging 
anc now ttiey Gl^ht be represented and taught^ explicitly and directly in 
orcec to avoid cne cost and other drawbacks of learning l^Pidlrectiy by 
^bservacior and practice. 

Related work on troubleshootlngfdebuggldg was exanlned and in 
conjui^cx^on with a logical analysis contributed to a characterization of 
t rouDlesnootinR/debugginsi probiens and probler^-soivlng processes that 
emphasizes tneir generality atross a number of technical f ields- and 
infori^al contexts. The analysis also suggests tkat debugging is a' 
^ fundaqental aspect of alnost all learning and probies sol^/lng. One 
-result of tne analysis v^s the forc^latlon of an inf omation-processing 
sodel of a general troublesnoot Ing/d^bugging strategy, which describes 
tne types of reasoning processes needed, sone of the factors governing 
selection of alternative processes in solving a probies, and an explicit 
control strategy. , 

Extensive exacinatian of a corpus of data fron students Learning 
conpater progranalng was undertaken, and soDe further lisited debugging 
data were collected froc both experienced and inexperienced prograrssers. 
Tnese aata are consistent with a hypothesis that expert debuggers do not 
necessarily nave superior general strategies-* but iastead that their 
experCj^ise derives froc specific and sosetic-es Idiosyncratic knowledge - 
acquired tnrough experience. Inexperienced. pregrarsDers lack t"&ls 
knowledge. Put In addition sose of thecj have a defective general 
strategy as well. In an attempt- to o&tain a rigorous characterizatlcm 
of the differences and def^cJts in the debugging strategy of the 
prograsQing students, an effort was Dade to apply a isodel-oriented^data 
analysis method reported in the literature. However, the eethod was 
unsuccessful for the data available and c^y have core basic litsitaClonS. 
As a consequence only mfon^l conclusions abouf the defective 
strategies used by^&^me inexperienced de^ggera could be developed: (1) 
tney are deficient In program testing an^so fall to find bugs; (2) they 
do npt Qollect or use available data about th^ effects of m bug to 
constrain their reasoning; (3) they have a low" threshold for attempting 
alnor apd soaetioes irrational repairs; and (4) they do not backtrack 
^ well ^roG unsuccessful repair atteepts. ^ * ^ , 

A snail-scale study was conducted to determine the effects of 
presenting a tutorial text* whicti explicitly describes a few general 



heuristics designed to correct these strategj^ deficits, to novice 
progracssers.. The data indicate a sar^inal increase the apparent use 
of soce of the heuri&tics by the pro^racser^* who studied the text 
compared to a group wno did not- ^ In addition comeents elicited froa 
the students were generally favorable to present ly^robleD-solving 
strategies explltitly, as .they weM in the tutorial. Hovever, the 
sticcess- of Che groups in solvioj^ debugging test proiiei^ did not differ. 
There yere severe! methodological limitations and problems encountered 
in the study vhich further confound t'ne results. I^ore general 
ttecnodological issues for stiKlies desip,ned to investigate instruction in 
troublesrf&otmg/d^buRging alsa becaoe apparent* One of the sost ^ 
it^portaa^- is analysis of complex problem-solving data: if the c auses oF 
differences in ability are to be specified in detail and ±t the effects 
of direct problecs^solving . instruct ion are to be-es^essed, then ic will 
t>e necessary to perfect cc^del-based data analysis n^thods* 
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!♦ Introductt^ 

♦ f* 
rne incr^asi^§ dependence of our society on technology is a 
pnenoDenon. Conplex ;sys tens continue to perforo new functions and 
beco!:^ t^re sophisticated. For exas^le, consider cheir role in oodern 
co£3=-ercial aviation. There are of course the oodern jet aircraft 
mcorpo^racing dozens of electrical. «lec|e4nic. and se^hanical syst'eos, 
i$uc tnere are also the netwofits of radar and comssraiiation Jysteos for 
controlling air traffic and the cosputerized scheduling and reservation 
systecs for coordinating flights and access to then by passengers and 
cargo, it IS difficult to isagine how the decands our society nod 
places on cossrerclal aviation cpuld be ^^sf led without these conplex 
systens. Such sy^cees nav become equally indispensible throughout our 
society* * * ' 

♦ 

Error or failure is always a threat when relying on a cosplex 
system, rne result aighc oerely be inconyttnience, as it would if an 
airline's reservation systec lost track of a passenger's reservation* 
-Ur, II could be^disaster, if, for example, an aircraft's radar failed in 
f lignt under tpnditions of poor visibility* Preventive oaintenance^nd 
repair df conplex systems is therefore an important concern- One 
respoDseVto the probleo have been efforts to develop better types pf 
technical data for both routine maintenance and repair procedures to 
,accoDpany complex systeos (Potter 6 Thomas, 1976'). A second, 
conpiesentary response. ^one with which this report is concerned* is to 
provide better training for the people responsible for testing and 
repairing complex systems, * 

If a ^stes does not operate as it should either during testing 



1 

or during actual- use — if the oil pressure warning ii^ht comes on in an 

aircraft » if ground radar incorrectly indicates the position /of 

aircraft, or if the reservation systea allows two passengers on the satae 

flignt to be assigned to seat iUA~, then a human technician mj»t be 

sumraoned to solve the problas of locating and correcting the cause of 

fne failure. Tr.is type of problec solving is r^ef erred to in different 

contexts cnpublescoot ipg or debugging * Tne objective of "good" 

trouDlesnoocmg/debugging is to locate and correct the cause of failures 

efficiently, wicnout undue cost of ir^aterials and titne. An electronics 

cecnnician does not want to replace several conponents i.n a circuic if 

ne nas reason to believe c^at only one of thee is faulted 3nd that he 

can Identify and replace just tha^t one in a reasonable atx>unt of tine. 

Similarly, a conputer progrann^r faced with a program tnat generates 

incorrect results wants to oake a relatively licited correction* one 

cnat does not entail receding parts of the program that perform their 

function adequately, 
♦ 

Expert troublesnooters , those technicians (or technical 

■ • 

consiritancs j who nake difTicult repat'r problems se^ easy and 

•inpossible'* ones only difficult, have always been highly valued aod are 

often regarded as artists, sincg their expertise is^ so poorly 

understood. Demand for their services can only grow as complex 

technology spreads. However, advancing technologies have introduced 
\ 

features such as built-in test systetns, modular aysten organization, and 
m-iniatijri«ation that make efficient troubleshooting of routine types of 
failures in even the most complex systems possible for technicians with 
more limited skill • Unfortunately , "many n'ewly trained technicians hav^ 
difficulty even with routine problems' and become competent only after 
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they have had considerable field experience. Thus, -aaintenance costs 
are hign and. in settings where there is ^/high-rate of personnel 
turnover, there tends*<o be a chronic shortage of competent technicians 
. . The research described in this report investigated the bases foi 
competence and expertise in troubleshooting, as se|n in the context of 
computer program debugging. The goal was to identify the types of 
knowledge tvecessary and useful for competent debugging aiTd to determine 
Whether new approaches to formal instructioo m-ight f3cilj,\at^ the 
attainnent of competent debugging ability by new programers . 

Troublesnooting/debuggin^ as a general aspect of probleo solving 

Situations that pose a problen of locating and correcting the 
cause of a failure are not liaited. to electronics, oec^ianics, and 
coTsputer progrataising and do not necessarily include coaplex technology. 
In some contexts, the parallels are straightforward enough to hav& • 
extended the common usage of the terns' "troubleshooting"^ and 

deouiiging". Management consultant's are often called troubleshooters . 
'-sing tne iMtHods of operations research, they locate causes of 
inefficiency in an organization . (corporalLions , agencies, etc.) and 
suggest corrections to its structure or procedures. The scope of these 
repairs is constrained by cost nuch as are those a technician can make 
In order to bring a device up to specifications. 

Less obviously, the behavior of a teacher tutorinaa single . 
student shares featur.es with that of a troubleshooter . In tutoring, the 
teacher asks questions evaluates the student's responses, and provides 
explanations In a continuing dialog .(Figure i). The purpose of some of 
his questions is to elicit answers that Identify specifrc inaccuracies 
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1 T: Do you^ ttrink it "Iain's^ much in Oregon? . - 

(Case selection: Oregon is-a paradigm case of fi7st order 
causal model of rainffl-l. ' . - 

Diagnosis: "ask foe a prediction abo^t a 'pcfiJ^icular case.) ' . 

2 S": No • • . . ' ■ 

{S's prediction' i'-s wrong) . ' * 

' * * f ^ i, 

3 T: Why do you think it doesn't rain much {n Oregc^n? 

/r.- ' . ^ \ ■ ^ ' ' ' . ' 

(Diagnosis: ask for* 4ny factors?) . - 

•-^ - , # • . 

4 ^: I'm not exacUy sure - just, hypothesizing -'it, seems to me'that 
the surrounding st^ites have rather dry climate, bu<: 1 really don't 
know anything about, the geography of. Oregon. . * 

(S's error' is due to a' proximity inference; ' S has no 
knowled^ of relevant factors) 

\ T: It does*' in fact rain a lot in Oregon* Can you' quess what 
causes the rai« therd? / ^ X % ^ 



(Correction: inform stU(Hnt. 
Diagnosis: ask for prior factors/) 

^ 'it. 

6 Sy Well, let me see - I h'ave a feeling that there is a mountain 
range nearby and the. ocean i.e. Pacific, I think -probably borders 
Oregon somewhat? ' ' 

(S -names '2 factors,^ but does, not mention their relatioaship 
to rainfall.) 

7 T: Yes the Pacific borders Oregop how do you think it is involved 
in the ^Jeavy raihfall there? | ^ ' 

(Diagnosis: T selects prior factor;' holds other factor: 
RUle: ask for intermediate factors.) 

4 

8 S: 1 haven't really got any idea - .well not quite true; I woul-d 
only be guessing. Does the air {moist ait) from the ocean somehow get 
blown ovex Oregon- and encounter a block of some sprt which ca-uses it 
to ri,se and codl? ^ - 

(S is missing three ste'ps'that ar\ in T's codel: !• why the 
air is moist, 2. why it^ is blown over Oregon^ 3. why 
coolitig results .in rain) ' " ' , 

" * ' * ** 

Figure 1. Annctated dlaic^ between a'bum^ tutor and stud^t* 
^ Frcd Stevens and CoUins, 1977^ 



or oq^issions in the sti/dent's know^ed^e, , Once these errors are 

detected-, the^ tutor may provid^ explatiaticns which he. believes wiU.-. 

correct them. -Alternatively, as. in the Socratic tutoring method, he may 

ask furthfer questions designed to prompt tfiT'student to reason about 

other knowledge he has -and thereby to co.rrect hims61f* (See Cdllins, 

i9Z6, for an. analysis of Socratic tutoring.) The- tutor is thus debuggiog 

the.stude'tjt's systenSof knowledge (Stevens and' Collins: 1977). • * 

♦ 

Troubieshooting/debugging= problems also ocCur Uj^^^^^ in a 
range of everyday contexts. Most coiinonly ;^ people arJH|d^th balky 
cars or househpld appliances, and atteiapt some limited troubleshooting 
CO avoid the expense and inconvenience of calling _a repairman or at 

% least to enable £hem_ to give hira a good description of the problem "if 
forced to call him. People also engage in informal debugging i^t^ 

"developing instructions". For instance, if someone -gets lost followiftg 
direction^ you gave them for. getting to y^r house,' then you engage iif . 
debugging when you determine which step of your instructions wer^ wrong 
.oj were exectjted incorrectly. If the 'instructions are lengthy, then it* 
can be effort to check them sCep-by-step from the beginning against a 
mental image of a map or'ofthe route yoii intended. Thus, to be more 
efficient, you might consider the locatiSn from which your friend' called 

^ou when he found himself Most and its proximity to points along the 
intended Jroute.' The analysis serves to limit the section, of your ' . ^ 
InstFuctions f^eed to exanttne for the error .^Th.is type of reasoning 
behavior resembles that of a computer programmer, gh6 uses the . 
characteristic's of a program's erroneous output- to suggest where he ' 
Should start tracing pro>?rao code. Other informal situations that 

■keq-uire debugging-like problem solving rang,e. from developing 



a new 



(Figure 3 continued)* 
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liVe learning' to. ride a bicycle: you watch someone else and then .climb 
on and try yourself. When you fall, you try to figiAre out why,, a^d ' * 
perhaps receive advice from a proficient bicyclist', such as 'iook at -the 
horizon, not the front wheeJL!" High motivation is required to learn 
troubleshooting/debugging in this way» since the frustrations one 

•encounters aw psychological analogies to skinnfed elbows .and knees. The 
^ instructor'^ method of facilitating the pro,cess ^is largely, empirical; he' 
tries to identify the ex^amples and exercises that result in better 

student performance on test exercises. 

The i^idirect approach to teaching troubleshooting/debugging does 

work satisfactorily for some students": after all, it isHihe way in- which 

existing competent troubleshdoters acquired their skill. Other students 

♦ 

: having "fallen of tfie bicycle" more times than they can bear (-or the 
educational system will allow) become drop-outs. In general however, 
the indirect, method is' less successful for teaching problem solving in 

\ technical, than in other sublets. The factor involved is the cost of 
resources required to generate exao^les ^nd to 'allow students to wo.rk on 
exercises. In mathematics or subjects based on mathematics, most 
problems can be solved with paper-and-pencil and the onl| demands are on 
the instructors imagination and energy and the' student's time. 

. Troubleshooting problems (and also design problems in engineering) 

require re^urces like, equipment and space, wTiich are Scaice commodities 
in most educational settings.' Since the cost of. these resources varies ' 
directly with the .amount of time used and number of errors jaade by 
students, there is an inherent pressure to limit student experience to a 
minimal number of simple, and less than realistic .^problems. The 
ll^mitations are oost critical ^or st<idents-havinf5' di^i icu'lty, .who fail , 

f * 

'•rig--'" 



to* have experiencesD^uf flcieot for learning tl\e required knowledge and 
so either drop-put or, fail* Even better students, however, may not get 

enough experie^ice tb becoae sufficiently coopetent by*'the time they 

,^ - 

finish formal inBtr^tion» Thus, new troubleshooters /debuggers roust 

■ : ^ - •'; V • 

typically undergo a peribd of on-the-job training, which is expensive ~^ 



-both becaifee their p'focjua.tivity is low and because it requires, the 
involvement of experienced- technlcistns . ^ - 



^ . * * 

A more direct appro^cji to teaching troubleshooting/debugging 

Oae appco^ch to impr^AiAng* formal instruction in 




^^^^^ 

troubleshooting/debugging is to r^uc6 the costs of th^ indirect method 
associated with providing es^amples to students and with operating and . 
supervising student problem-solving laboratories (Finch, 1971)* 
However, there is an ippar'ent paradox in the indirect method that could 
indicate a need for a" substantially different approach t9 instruction 
fpr some students. The paradox is th^t the learning by example and 
trial -and -error experiend^e required by the indirect method 'may actually 
presuppose the very ^roblem-solv^ing strategies the student is attempting 
to learn, (recall the analogy of c'titori-ng *as ''debugging the student"). 
In effect, learning by the indirect approach requires the student to 
debug his strategy for hov to debu^*. 

Since people do learn to debug by observation and practice, no 
real paradox exists. C^|^ly, sophisticated strategies must evolve by 
bodtstrapping from a primitive learping mechanism, which we is 
effective, ^though legs than optimal, for •inductive learning' in simple 



For scientific professionals, the lattet^^ars of gr^duatze 
education involve research experience tteat seryea a similar function for 
dev€^loping* problem-solving skills, * *' 



contexts. Stfudents' ±n technical disciplines bring to the classroom 
debugging scrategles, pf varying Effectiveness Which they have Irvduced by 
monitoring the-ir attempts. to solve the types'of informal everyday 
*'t roubles fi*J>tipg/debiigging problems we Mentioned earlier. Some of then 
-may alread)^ l^ave eff^ttiye general strategies and only haye to learii how 
to apply th^ in -a/new problem domaia. . The indirect method works for 
them because their debugging strategies help thea to learn efficiently 
from their ^experiences} they are proficient at^debugging their own 
knowledge. However, those students with liT^fective and inefficient 
initial strategies encounter a bootstrapping problem because efficient 
learning by ijjduction presupposes some of the same strategies as 
debuggingr Therefore; another approach to instruction in 
troubleshooting/debugging, whiqh would be most advantageous to students 
of lower initial ability, is to try to teach EK>re directly and. 
expl^fjitly the general strategies that students develop when they 
understand example^>and try to solve p^oblen^ themselves". Such 
instruction could help^ Students acquire an effective strategy for 
troubleahooting'/debugging ©pre rapidly and is^rove their general 
capability to learn by the indirect method to troubleshoot in a 

particular domain^ ' . ^ . — 

' _ "» 

' The are two aspects to developing an alternativfe^, more direct ' 
^proach for teaching troubleshoot tog/debugging. First, the strategies ' 
that students learn by observing competent .problem solvers and by* f • 
solving practice probleias must be identified aqd articulated ji.e.r 
represented). -Second, a Citable pedagogy must be ^prtrolated- These / 
goais are not -necessarily Independent/ since p^agoglcal decisions cUn 
depend on the way the knowledge is'tepresented and conversely, choices ' 



ataong alternative representations can depend on features of' preferred or 
available teaching.. mfetbods , ' 

s 

In the remaining sections o£ this report, we wilJL discuss the 
J* ' . , / ' ^ , 

ideas' of orhers and ourselves about the' nature of the> ^ ' 

troubleshooting/debuRging process- We will describe our obsei^ations of 
computer program debugging behavior which bear upon, these conceptions, 
and which also suggest the knowledge deficits tha't^cause soise 
inexperienced prograzamers to have difficalty with even sinple rfebuggin^ ■ 
problems. He will conclude by presenting the results of a 'study 
designed co' investigate whether such deficits sight be corrected by 
direct instruction. * ^ ' 



• / 
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!!• UuderstandiPi^ the troubleshooting/debuggiog process 

r ^ 

Difficulties ia studying troubleshooting/debugging ^ 
One reason that troubleshooting/debugging (and ot}rer types of 
caof^iex proW^ solving) ar^taughx indirectly is that tt is difficult ^ 
to gather the data needed to devel<?p an espirically-based raderstandiog 
9f cne problen-soiving process. Tliere are probiccis of observing a range 
of episodes and of the ^server not interfering with the 
croubleshooter's behavior. Si^le probieiss esay be solved in tsinutes 
during a single "sitting", 'while CKjplex problec^ cay be solved over 
days or 'wen weeks (e.g., the debugging problens faced by systea 
progranners on large cocputer systens). Thus, It is =uch t»re difficult 
CO observe th6 solutions to problecs at the core difficult end of the 
speccrun. In any episode, there is the problecs of observing the 
troublesnooter without causing hia to depart froa his normal procedures. 

A general limitation in studying troubleshooting episodes is 
that.isuch of the troubleshooter 's tise is spent in periods of thinking, 
during wtilch there is no^ overt behavior to observe. lypicailyTit is 
dlfficult^to infer what the problen solver is tti4nkiog froa the behavior 
oDserved prior and subsequent to these quiet. periods. Post hoc reports 
(e.g., "Tell Eie how you solved that problea") tend to be edited and 
incosplete, appearing as idealized accounts tdiich frequently conflict 
with observed behavioral data. Mortf gener^ self-ceports CTell ae how 
you troublfeshoot") oay also be contradictory and incoftplete. Thert is a 
truiSQ that being an expert at doing soaethlng does not necessarily 
i^ly being able to intrpspec; on how one' does it. 



11 

IS . 




Troubleshooting/debuggin^N-afififa^to be an activity for which is-noV easy 
for Bost exper^ts to desctibe^ their reasoning in either particular or 
geoeral^teips. "nie ^fictitious dialog in Figure 2 caricatures this > 



. * inability. 



>• ' * There has also been a difficulty in analyzing ^d organizing the 
behavioral data and self-reports that 'can be obtaiped^ Prior to the 



V, 



devfeLopoent of infomation-^rqce^ing aod cybernetics there w^s no 

adequate fpnaali^ for describing processes — i.e., to represent 

J^roc^ural knowledge— and thus for interpreting and, integrating^ setfe of 

observations in order to develop and. test hypotheses relat^ngfhe 

knowledge used. by troubleshooters solvers and differences £n 

>leshooclng episodes. While natural language has been used to 

^^esent propositional knowledge fros the earliest titles, it is a poor 

Dedius for expressing cooplex procedural knowledge* To convince 

$ I 

yourself of this consider the typical cosprehensibility of the assenbly 

and operating instructidns for various devices* Usually', one renains 

uncertain of his understanding until the device works (i.e*, the 

instructions ate "understandable only if you already know the process). 

One apparent weakness of natural lan^yage for describing processes is 

* 

its awkwardness and acbiguity for expfe&sing complex conditional 

relationships between events. Hojre generally, ifi naxural language such 

< 

of jtbe knowledge being transmitted by the sender is iiapjicit and isust be 

t • K 

inferred by the receiver* The deisands for decoding the li^licit; 
knowledge be sore severe for procedural than for propositiopal 
knowledge. (Try to generate a sufficient description of how to drive a 
car that you can' feel conf ide^nt will be understood without questions by 
sos^ne who has never driven one*) The liaitations ^f natua^l lahguage 
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'*Houi did you knou the trouble oiias 
in the ^iiiitch?" 



**B8cau5e it uiorked intermittently 
^luihen I jiggled the suitch. ** 

. '"Well--- coulan/t it jxggie the'^iref^** 

"Houj do* you ++knouyfr all "that?"* 

'•It's -^^-^Gbv'xcose. ^ 

"Uell then/ uihy didn't I see it." 

some familiarity. 



"You have fo h 



"Then it^'lg ++note obvious, is it?**' 

Fictional 6ia2x^ between an expert' tTOubleshooter <E)' and 
an oheetwer (O) caricaturing the expert's difficulty 'in 
articulating the soifrce of his expertise. Frcoi Zen and - 
tne _^.of Motorcycle H^tenance. p. 135'(PLrBig7"l97lt). 

c . - . „ .. . * 
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map-be partly responsible for the difficulties that problem-solvers 'seen 
to have Introspecting: besides^h%ir difficulties in realizing how they 
troubleshoot. they may not be able to articulate what they are aware of. 

Thuk. understanding of the troubleshooting/debugging process his 
been hamperedV^oth by difficulties in sftking rf:omplete, valid- 
observations and in systematically int^xpreting the data that can be 

♦ 

obtained* 

^gf omation-processin^ podels ^ ^ 

-,;^er the past twenty y^srs researchte^s, i^lnfora^on-processing 
psychology concerne^ with understaQdiig intallige^ huiiian behav^ior and 
tho|e in artificial intelligeQce (AI) interested In developing 
'intelligent" computer systexas have developted new fornalisas for 

--representing knowledge. Semantic networks 4Quillian. 1-969; ji^pds, 
i975). production systems (Newell, 1975), procedural networks 'f Brown, 
Surton, Hausnann, Goldstein, tluggins; & Miller, 1977^-^erdoti, 1975). 
logical calculi (Nilsson, 1^71), and process ^raanars (Killer S 
Goldstein, 1976a; Woods 1970) are the new "languages" used to represent* 

the declarative and procedural knowledge underlying intelligent behavior 

% r • ' 

in a range of tasks . ^ese fonaallsss^ave enabled the development of • 

<- 

sufficfent ("strong") computational aodels tor certain well-structured 

probleo domains, such as logical pr'oot, gaaes, and pUzzIes. Therl are ' 

sow computer, programs that crfg^olve such {Problems as veil or better 

than most human problem solvers. Strong computationkl models have also 

b^en used to simulate human problem-solving tehavior.c including its 

• ' / . f. , , 

variability and errors, in an analysis-by^yrvphesis agproach to 

interpreting behavioral and introspective da"ta '(Newell & Simon, 1972), 
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Beyond their application in autoaat^a problem solvers, "the 
knowledge repreJeqtatloos that" have been developed provide a framework*^ 
for analyzing observations and for articulating partial model^ of less . 
well understood types tf problem 8<>^lving like troubleshoot^ing/debugginR. 
That is, even if ^t is not yet possible to' write a general prograa to 
troubleshoot faults' ia circuits or one to debug other programs. It say' 
be possible represent the top-level organization such a prograa would 
need and soae of its oore specific data-structures and procedures. Su«h 
"weak" oodels are a basis for directing attention to aspects of the 
process -that are not' yet understood and "their logical relationships to 
, those that are and for interjJrfeting new data in order to expand our 
. understanding. 

Over the p&st several ye^s, psychologists and computer 

scientists working the the field rffAl at MIT have conducted research on. 

information-processing tsodels of prp^raBalng and debugging. As a 

conse,quence of their work they have come to adopt a view that debugging 

is a fundaoehtal aspect of most, if not all. c<hj5>le:t human learning and ' 

oblen solving (Goldstein. 1975; Miller & Goldstein/ 1976a; Papert, 

^1971; Ruth, 1974; Sussnan. 1973). The position is based on their 

.inforraal 'analyses of human Brograsaing behavior and on their attempts to 

dfevelop "intelligeiM:" prograas for writing prograias and for solving 

other type? of problems. People learning to prograa and even. 

experienced prograaaers designing prograos knowingly code and att^C to 

•. execute prpgraas that are Inadequate. They say be unsure about the 

effects of a particular^ contruct or of tbe:lnteraction of faiaillar 

. constructs in combination. When the- prograa falls, hy reasoning from 
*** . • 

the way It failed they can modify it to function .correctly. As a sia^le 



J 



exaopie, a statistical program na^ involve printing a table with a 
complicated format that depends on the parameters, of the data ta be ' 
analyzed. The prograooer writing the program may have 'difficulty 
calculating the format parameters. needed' to align the headings and' 
. entries in the table. He may therefore proceed by estimating the' format 
and. then executing the program. The .ercors he observes enable him to 
modify his original estimates to produce a correct format. 

This notion of the generality of debugging goes beyond", our 
earlier comnants about .the range of situations in which Jebugging-like 
behavior is requiredj it says more strongly that problem solyers 
consciously: create debugging problems for themselves as part of a v. 
general planning strategy.' l>^bii^inj is seen as^ a natural complement to 
design in the process of planning and implenenting a program. Either 
because it is more efficient or because human inbrmation-processing 
capabilities (e.g., ia "working memory") limit the Oomplexity of the 
design process, programmers implement programs with ^ expectation that 
they will have to debug them- i.e., debugging is not necessarily an ^ 
afterthought forced on programmers. 

There is an alternative view of debugging that it is a 
regrettable outcome of poor^design and that programmers can and should 

strive to eliminate all debugging through rigorous de'sign/ This 

*■ - ■ "O ■ 

position is popular among advocates of "structured prograaaing" (Dahl, ~ 

Dijkstra. & Hoare, 1972). .We disagree with this vifewpoint. wii;ie *' * 

rigorous ^itial top-down program design" is certainly c^esirable, it is 

unrealistic ceo demand and expect flawless design, for complex, innovative 

programs. Our own informal obseVvations of skillful professional 

programmers indicate that; despite conscientious efforts at top-doiii ' 



ERIC 



9^ • • .16 



) 



24 . . 



design^ they inevitably scart iii?>leaentinj; and test.ing^ programs before 
the' design is Complete. It' seems that there^are too many complex 
interrelationships in taost prograds and that they can be understood and 
impleaented aore^ easily by debugging than by abstract logical analysis. 
From the prograaaer's perspective, there is a strategio tradeoff between 
the -costs of design and debugging such that;, it i« most efficient to 
integrate the two so as to minimize the the mi^imm complexity at any 
Roint. " ■ • 

, From their stydies of programming, the 'researchers #t MIT have- ' 

generalized the constructive rode of debugging in learning and problem , 
solving using the following logic. A con^uter progtaa Is a' ^ 
representation of a plan, a sequence of legal operations Id an 
envirotmettt that when executed will acconylish a goal (i.e., solve a 
problem). For example, a sorting program'^^is a plan for accomplishing 
the goal of arranging a list of values in a desired order from an 
arbitrary initial order. However, writing and executing a prograa is 
only one way of expressing and following a plan. Plans were developed 
»apd. executed by people to solve problems long before computers existed 
and have beenTembodied as mechanical and electro-oechanical sy^teas. 
Programs are Just a general way of representing plans. That 'is. why 
progr^ (Jan be used to •Bimulate some^of the behflv'ibr of people and of 
mechanical and electronic systems. 

Plans then, like programs, may also f irst b& -f o^^lated with 
some ignorance of whether particular actions will be effective. If 
execution of the plan proves it Inadequate and if the plan is |:o be used 
again^ thenr the inforaatioa obtained -f roe, the failure can be' applied to 
modify the plan. • However, even if the plan was for a unique 'problea and 
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will never be used again, debugging the plan is useful. In designing • 

new plans, parts of old plans for somewhat t*el3ted problems may be used 

*. « 

and so a 'library" of correct plans can iielp the problem solver* 

• ' ' i ' ' ^ 

Furthermore, one can- sefe that there must be "plans ^.r planning"— 

general strategies for^kingr design and debugging decisions in planl\ing 

solutions to- particular problems (e-g*, whether to^ynthesize part, of a 

design or borrow it frr^m a design in one's plan librar:^. pfen failures 

provide feedback that can be used to debug not only thV fSolty *plan, but 



also the strategy used to design it in the first place, 

Sussman (1973) developed many of these ideas about planning and' 
d^u^ing in the course of formulating a computational isodel called 
HACKER, a program that solves problems in the paradigmatic 
"blocks-world" domain." ^iven a- problem oj rearranging some of the 
blocks on a table, -HACKER in its naive starting state designs a solution 
of simple actions (pick up, put on) .J^ep ending on the problem,' its 
iniUai solution may succeet^or fail (where failure is defined by aotion 
sequences that are redapdant'or impossible in the blocks world). In 
case of failure, HAjfKER works to debug the plan <nof always 
successf ullyj • It also stores information about correct plans and about 
bugs that it^ can use in designing solutions f or -s^Ubsequent problems.^ 

Mark Miller* and Ira Goldstein at MIT 04iller & Goldstein, 1976a) 
bave attempted Co formalize the relationship betwe^ design and 
debugging in problem solving using what they call planning gramaars . J# 
which are representations of design and debugg^ing strategies. Employing 
both context-free -graonars ami augmented transition nett^prks (XTN) 

' . r\ 

Se^ Sacerdoti (1975) for an alternative view that cot^ebt plans 
can be lapleaented in incrementat ^ages of design and execjutidn^ without 
, debugging* / . \ . 
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(Woods, "1970), they have written systems of rules that« describe the 
process of creating»and. execu'ting COGO grajJhics programs (Ftgure 3). ^ 
<pey have proposed that pllnning. gramma!<s can serve two functions: 

(1) inte'rpteting and comparing the behavior of different programaers and 

(2) developing •■intelllgtot" afg^ems for assisting programmers in 
designing and debugging xheir programs, They-hav'e explored the second 
use in their SPADE systenf (Miller & Goldstein, 19-76b)", which records a ' 
Frogrammer's^lanning decisions with respect to a" planning grammar. The 
record. is used to advise the programmer of conflicts and omissions in " 
the structure of the program and o! his options any point in the 
planning process (Figure 4). " 

A sener-al characterization of troubieshooting/debugeint. -Droh1emfi 

Our examination of research on troubleshooting/debugging has led 
us to formu-late a characterization of troubiOiooting and debugging; 
general co-a range ^of problem domains. 

We define t roubreshoo.^ing/^debugging as a type of problem solving 
focused on either an abstrao^plan or a procedural system . A procedural 
system is a physical entity that Embodies a plan and can execute' It -to - 
accomplish its goal. A characteristic of pl«ns and procedural systeigs 
'Is that they can be represented as hierarchies of functional subparts, 
each subpart having^ specitic role in achievement of the overalL g<(al^ 

For instance,' a plan for buUdlnj? a table tncludes subplajis for ' ' 

/ 

obtaining a design, obtaini'tig materials, asseabling the wood and 
hardware, and f in.ishing the assembled taMfe. , Each 6f these subplans 
ftonsists of smaller subplans. The plan for obtaining materials might 
include -^ubplans for borrowing a truck, s^l^ing a Id^ber supplier, 
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Figure 3. Miller aii4 Goldstein's {1976b) content-free 
for planning and debugging prc^^rams. 
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(Figure 3 continued)* 



DEBUG " <CDIAGNOSE] + tREPAIR]>' 



DIAGNOSE 
TRACE 
SELF-DOC ' 

. REPAIR 
ADD-PAUSE 
ADD-PRINT 
ADD-TRACE 
EDIT 

• RUN 
ADD 

•DELETE 
CHANGE . 

* 



-> <ASK t TRACE I ■er^r"/ 
-> [SELF-DOC*] + RUN* 
-> ADD-PAUSE I ADD-PRIMT I ADD-TRACE 
-> "print definition* | ■print value' | ■print file^l 



-> <RUN I EDIT I SOLVEX 
-> ADD 



•> ADD 



-> ADD ■ _^ 

ADD I DELETE | CHANGE - 
•> "run statemtnt of code" + "response" [DEBUG] 
-> "add stateaent ef code" ^ "response* + [PEBUG] 
-> "delete stateasnt of code" "response* 4- tDEBUGJ 
"change stateseftt of code" + "response* ♦ CDioUGJ 



SPADE-0 Scenario fpart a) 



The preliainot^ ver^on of SPADE U little mre than a bookkeeper for th« 
user's subgoals.- It ifijcarreiittg Jtting iapletiented bff assignitis an interpretive 
procedure to each gramatt,ijs(tl prtaittoe. 



la. What is tMe naaa of your. top levej. procedure? . 
lb.' >W \ . ■ 



yZa. Rale for WW is: SOLVE— > PUW ♦.[DEBUSJ. " v 

Rule for WH-1 is: PLAN -> IDENTIFY j DECOMPOSE I REFORJIUUTE. 
What noW?, ' • 

Zb. >DECO«POSE . , ' c 



3a.. Rule for WW-4- is:. LINEAR -> SEQUENTIAL J SET. 

What now? 
3b. >SEQUEffT'lAL - 



4a. Rule for*W»-5 is: SEQ -> [SETUP] ♦ WtSi ■¥ [I»TERFACE3>» * CCLEANUP3 

Do you wi^b to include the optional constituent SETUP? 
4b. >No ^ " 



5a. How nanjt occurrences of <r!AIN 
5b. >3 



'ACE}> do you want? 



6a. Do you wish to include ttie optlomal constit»«it CLEANUP? 
6b. >LATER " • ' . ^-t 



MTER 'ts q cwasand undlirttood ij/ BfADE. It aeons that the user wonts to 
terminate this path, butlthat it mst berectmencei later, $PADE Keeps track of 
confunctive goal^, reminkng the u&er of the 'peniinj goal list if asked, here it 
records the informtionj^qaired to return to, and eventually complete, the 
applicatioif of tAe^SfQ. role'. It can then go'.on to other goals. " 



Figuie 4, Di^Lc^ with p3^rajaaer*BY«Essistant based on the grasaar 
sfacjwn in Figure 3\;— FiW I^ller and Goldetein, ig76b. 



* <* 
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(Figure h continued) 



SPADE-0 Scenario^ (part b) 



7a. Your pending goals are •..and WELL. Vhat NOW? 
' ^ 7b. >PbLt 

*• * I - 

* 0 

I • • 

SPADE eppties the SOLVE rale recarsivelg. 

8a. Rule for POLE is: SOLVE -> PLAN ♦ [DEBUG! 

Rule for POLE-1 is: IDEHTIFY J DECOMPOSE | REFORHULATE. 

What now? 
8b. >IDENTIFY 

9b. Rule for POLE-2 is:' iDENT^fY -> PRJHITS^ | DEFIHED. 

What now? . ' . ,. 

9b. >PRIBiTIVE 

10a. Rule for POLE-3 Is: PRIHITIVE -> VECTOR \ "ROTATiroi I PEKSttTE 

What now? . - ■ ^ r 

lOb. >VECTOR . ' . , 



>FORHARD 100 

11a. POLE doDpleted. Pending goals are: ROOF, WELL, and W-17 (CLEAHUP of 
WW).' What now? 

lib.. >WELL^ ' ' » - " 

4 " ' - 



;. . . 

, ^ I2a. Rule 'fop WELL-1 £s: PLAh' -> IDEHTfFf I- DECOSPOSE f REFWHlttATE. 
What now?- 

^?b. >DECO?fPOSE } . .■ ' ' ■ 

Here toe have subsUtuted a' graemr tshich contains rales for conjunction hat not 
repetition. This, allotts us to illustrate the mnnerM uhich SPJWi awoiis - 
interrogating the user ehen no actMal iiedsion tj reqnlrei* 

Jjli. Rule for WELL-4 is: cfeoHIPOSE -> COWJUNCTIOH. 
(Forced.) ■ _ ' 

Role for.WELl.-5 JS: CO?IJIWCnON -> LINEAR I HOHLWEW 
What now? 
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selecting a hardware suppWer , aniTswtieduling^he trip to nake the.' 
purchases. ^ another exac^Ie, an electronic power supply consists of 
subcircuits such as aepllfters, vqltage^^regstlaiors, etc. In tumf^each 
of subcircuit consists of oore basic subcircuits, and so on until the 
ieyel of prinitive coaponents— traiisistors, resistors diodes, wires,-- 
.etc. Siailarly, a computer prograia will typicaily have staprograns for 

input, output, initialization, sorting, etc. * ) ' • 

' 7 ' 

A fe^ture^of such functionally d^fit>ed hierarchies is that the 

subparts at each level are independent in that each is a ''black box" 

J ^ 

froD the viewpoint of the Qthers; it doesn't ^i-attet now each does-vhat 
it does, as long as it fulfills its Tt)le in atcain|ng the overall goal. 
For instance, in^ asse:^llng the table, the details of the -subpla^ by 
vfiich the caceriaU were obtained are irrelevant as. lonsi as the 
naterials are all there TOen. the* assembly subplan is executed. 
Siailari^c, structurally d fife rent b^t functionally equivalent voltage 
regulator circuits can be interchanged in a pover supply and differ^t 
sorting algorithms qan be interchanged in a program-** 

The Mparts at esch level of the fimctHfaal ^hierarchy have a 
xeleologicai structure. In tb^ 3i^lest^ linear structure, the act;ioo 
of each subpart depends directly on that of one other subpa^ and 
affects directly one ot^er sisbpart. Of c^^urse. the action bf a subpart 
can indirectly affect all^the subpart^ subsequent to it in the' 



The relationship between subparts at a level of the hierarchy 
can be i^re coep^lcated than this, since it is possible for tb^ to be 
funcabnally discrete, bu|: still shai^e physical structure. For exa^le, 
two subprograms for input and output say share C'call") a 
type-conversioij subprograa ajb a lover le^el* This oveijlap Is^incidental 
vln that shared structure can be replaced by redundant cc^ies/^W 
i^rtant In £hat a defect in the shared structure say affect thSv^ 
ftmctlon of. all sutierordinate parts* 
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♦ teleologica^^^uctire. More coaplicat^ structures liave miltiple 

interfaces and feedback ^paths between subparts. When a subpart* contahs-^ 
a fault,' thCT its action will b€( incorrect' for at least soae of ,tfie 
possible actions of ita^iately prior subparts. Its faulty actions say 
inhibit- subsequent subparts froa operatingSand teraiaate the 

oper^ion of th5 entire plaa/sy8tee-6r say propagate through the«*and ■ 
distort Che actions of the plan/systrai. » 

Troubl^hc-otiag/debugging involves resasoning about the actions 
of the. plan/systes and its teleological structure "at each lev^ of its' 
fuactiooal hieracchy in order to locali^ the fault to a siniam nusber . 
of subparts (ideally one) at that leve\. The actions and structure of 
the' suspect subpart (s) are then used to localize -the -faiiit at the nect 
. lower level of the Eimctionai hier^chy, and 50 on until 'the cost of 
repairing a subpart (s) is less than the cdst of further localization-' 

' ' Expected cost plays several role^ in debugging.. It not infy 
det^ines the level at which repair is .attempted, but also ^rves to 
order logically equivalent debujing actions. Cost depends 05 how the 
structure of the pian/systte affects sseasuresents of the actlons^f = 
itbe ability to repair a particular subpart. It also is detetkined fr^ ■ 
the debugger's idibsyn^ratic experiences. For exagple, %.t a cir liiles " 
unevenly, an experienced ssechanic Eay jar the" carburetor In case a piece 
of dirt ls.lodg«i in one of tlje «saH Internal passages before he has 
done any tests on the i^ition tialag,, spark plugs, or eng^ * |^ 
cbi^jression which sight logically detersine that the proble. U acofiUy »' 
.in the carburetor. His att^ted repair in this case is Ine^eniive 
enough to allov tes«y of a hypothesis ^^oped by induct>4 ("une4i 
idle has in past experl^ce been ^sociated with dlii In the . 



carburetor**) rather than by deduction ("the observations that have been 
Bade iogically detemine that the probiea must be. in the carburetor")/ 
As an illustration of how cost thresholds enter into reasonipg 
^ debugging, consider a sla^le hooe troubleshooting problea» Suppose 
you wake up during the night and decide to go to. the kitchen^for a 
snack. When you oove the switch on your" beside lasp to ^e "OK" ♦ 
position, Che larq> fails to light. Given that you^are isotivated to 
^discover the cause of the failure and, if possible, effect a repair 
what vould constitute 'an effective and efficient tack. If there have 
been previous plroblens with the lai^ th^t you have traced ^o an 
intermittent short in its switch, you sight operate the switch several 
tis^s in an atteept to "unahort" it tesporarily, Tnat.is, you night 
identify the syinpton and irraedlately recognize a possible cause that 
your experiejice suggests say be ©ore likely"- than other possible causes 
and that has atf inexpensive (if ceisporary) repair, if y cm had no such 

reason for suspecting the svitch, then you oust reason, about the circuit 

\ { 

(procedural system) I that contains the lacip* The lat^ cl^rcuit has a 
sis^le linear teleoldgy consisting of the external power supply to th^ 
house, a fuse or circuit breaker, . the^ wall outlet, the la^ piGg and 
cord, tile laep switch, the light bulb, dnd several iatervening"^8ection8 . 
of wiring. The light* 'bulb will not light {the initial syaptoa you 
observed), if there is a fault in any of the coi^oneats prior to it. 

One aipect of an effective general troubleshooting/debugging 
strategy is to make observations that, g^ffen the structure of the . , 
systes, are logically sufficient to exclude or ^Include subparts f^oia a ^ 
location hypothesis , which is si^ly a description of wbete^he f aidt 
couTd possibly be Ipcated in the systea. ^e. actions' of ai^,stibpirt in , 

• . ^' .26 34 • 



a Unear teleological structure can serve to-refine the hypothesis in 
one of two ways. If the actions are normal at point A, then the fault 

« 

must be in a subpart subsequent to that point; if the actions are 
abnormal at A, then the fault oust be in a prior subpart. Thus, if all 
"subparts can be observed 'with equal cost and if the debugger' has no 
'•special knowledge relating the observations he can sake to the 
likelihood of possible faults (e'.g.', the lamp switch has an interalt^ent 
short, bulbs fail 5 tiaes as often as fuTes. etc), then an optinal ' 
strategy is to nake observations that will repeatedly halve Jh'e scope of 
Che location hypothesis until the fault is isolated to a single subpart; 
this Qiniaizes-the expected nuaber^-of obaervatioos that are required to 

localize to a single subpart. Thus, .in our exa^le', because the wall ' 

■ ^ " " 

outlet is near the giddle of the ia!^ 'circuit, • the first observation 

would be to see If the Isap is Riugged in and, 1^ so, whetheV another 

electrl'cal device connected to the sane outlet operates correctly.- 

Suppose 'that the laisp proves to be plugged in and furthersore 

that an electric clock is plugged in. at the saae wall outlet so ,that you 

can easil3> (without g»tting out of bed) observe whether it is still ' 

operating. If it is, then the fault can not be in the house power 

supply, the f^ase, or any of the /onnecting wires prior to the wall 

outlet. If the clock is' not working, then the fault is In one 'of those 

subsysteas (barring two independent failures in the laop and clock). 

Ut's assuse the clock also is not working. Breaks in house .wiring are 

ordinarily uncoaaon, and so it is aost likely that either the power ' 

s«pplyk has failed or the fuse has blown. Since the fuse box is in the 

baseaent, it is "costly" to check /relative to looking out your window 

to see if the street lights are still working'. If those lighttf are off, 
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then the power has failed.. If they are on, then you can replace the ' 
fuse. )If that doesn't solve the problem, then get your snack, go back 
tx> sleep, and call an electrician io the morning, lecaiise the problem is ' ^ 
in the internal wiring of the house.)''' 

This is an efficie&t way to ^^late the "fault, though given 
slight changes in the situation other solutions sight become better, 
. For instance, if your bed is next to a window, then the" easiest ' ' 
observation tp sta^ with (behK£^ooking at your clock) aight be to 

look o'ut at the street lights. Of course, if they Sre on, then you know C * 

only tjiat the power supply is intact-only o'nfe subsys-ten has been " 
elininated conpared to the three or four elialnated by checking the 
clock wfaether^it is working or notV, .The strategy for making ' 
observations seems general in itself /^t^nTTthis episode requires 
knowledge of the lamp circuit and is affected by idiosyncratic knowledge 
and by parameters of the situation that determine the costs of making ' 
observations and repair^- " ' 

Representrg^ a' s^^gy for troubleshooting/debue^ in(^ 
^ In order to understand more precise^fiiow different types ^of 

knowledge are used in troubleshopting/debug 'ng, we developed aW 
in£a,6ationr-proce8Sing model for a gei^ral debugging strategy, like the 
one;illt.gtrated in'the above example. The mod^l is^general in the sense 
that it Is intended to descrtfi-e the overall structure of successful. , ' 

^ debugging episodes by differeirT^lviduals foir different problems in a 
range of subject domains. Variations In the , structure of any episode - 
are due to characteristics of domains and problems and differences in 
the dooain-specifis knowledge of individuals, 'ihe model identifles^e 
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points at which these factors produce vari^tiQM in problea-splving 
behavior. It is a very "weak" model In that is far from a sufficient 
computational aodel of troubleshobting/debugging ,in any doaain. 
However, it is intended to be a logically sufficient description of a 
top-level organizational structure for a strong aodel. Our Eodel draws 
ypon prior research on debugging mentioned earlier, particularly the 
planning graHBar8_ of Miller and Goldstein (1976a, fa) 

The aodel is a representation of procedural knowledge and we 
have ctiosen to express it here as a type of procedural networlT (Figure 
5). We considered, but dismissed, the possibility of using a preduction 
system formaliss to represent the aodel. The "prioary factor in this 
decision-was that production systeas hide the control structbre of a 

♦ 

p^rocedure by distributing it across t^^e individual productions* A 
second factoT was'' that production systess incorporate seoantic tests at 
every point .in the control structure— they presuppose that all 
pKacedures are invoiced conditionally— while we found that we wanted to 
identify both conditional and unconditional calling relationships^ The 
procedural network overcoaes both of these dif f icuj^ties,^ Firsts i^ ^ 
explicitly representar the overall control structure of the iio4el* 
Second, by annotating the connections between procedures, ^ondltionaX 
and unconditional flow of control- are conveniently distinguished. An 
ATN fons^iss also has" a natural way of distinguishing conditional and ? 
unconditional paths of control, but we found it soae^at less heuristic 
for coammicating' the entire top-down structure of "the aodel. We want 
to eaph^ize that the node! could have been represented as a pro^ctlon . 
system, but with ifis^ ef^ciency^nd cor^rehensibility, 

The .notation in 'Figure ^5 requires some explanation^ Each node 

■ .'i 
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DETERMINE- 
OBSER VAT IONS 



RECOGNIZE- 

. . BUG 



MAKE-BEST-^ 
OBSERV^a'ION 

I " 

21b 



MODIFY-ACTION- 
DESCRIPTION 



/C HARACT ERIZE 



TEST 



2{o / 



DEBUS' 



MOOlh'- 
LOCAT ION- 
HYPOTHESIS 



REPAIR - M 



Figure 5. Procedixral netwesik for.tbe top-level structure of 
a general strategy for troublesjiootl^/debugging. 
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in the^ procedural njetwork is the naoe of a"proce<&te defined by its 
function. An arrow from one node to another indicates that the 
procedure at the tail cllls, o'r passes coat^l dovm to, the procedure at 
the head. If a node has iwre. than one arrow emanating froa it, the 
calls froa it are alwa^ made in the order denoted by the integers" 
labeling the arrows. Solid. arrows represent uncomlltional calls, while 
dashed arrows, are calls aade only if sooe" semantic tests are first- 
satisfied. Each dashed arrow is labeled with a letter. Table 1 • 
summarizes the semantic tests for each of .these labeled dashed arrows' in 
Figure 5^and tists the global registers and d^ta structures used by the 
raod^l,^ In tracing flow of control in the network, the following 
convention is in force: vhen a procedure finishes {when all its 
subordinate procedures finish) it passes control back up to the 
superordibate procedure that called it; that superordinate procedure, 
then calls its next subordiiiate procedure^ (if any)* 

Since a generaj. strategy for troiibleshootiacg/debugging Is a plan, 
for solving problems, its representation as a procedural network can be 
viewed in the same way as the plans and procedural systeas the^strategy 
can be applied to. That is, the levels of "the network are levels o'f the 
strategy's functional hierarchy. The hierarchy Is incoaplete in that it 
do6s not extend dowi to tlie prlaltive procedures needed to. solve 
problems in any specific dondin. The teleological structure of the 
hierarchy is cdoplex {not linear) and iA represented In part by" the 
ordering on arrows emanating down froa a node. A second part- of the 



4 , ' 

Since* isuch of the coozminication amon^ procedures is by global 
structures, the recursive procedure calls in%e network lo aost cases 
do not increase the aeiaory demands of the aodel beyond those that would- 
be iiaposed by iterative calli* ' * 



^'Table 1 

Of Global. Data Strudnires and Itegisters * 
and of I>roced\ire Invocation Semantics of 
. the Troubleshooting/Debugging Model 
Illustrated in Figure 5. 
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Data structtires and registers 
ACTICa-DESCRIPITOR 



list of i)rc7positions describing, foe relation between 
observed- and normal plan/syBtem actions. 



LOe/fflCS-HypOT^ISi description of parts o|^Wsysteia vbere a fault 

aay possibly exist, y*-^ 



SRBOR? 



TEJE if no error has been detected or if a retjair 
had been made' and not yet tested; FAI^ otbsivise. 

u nidl TTensional value that is a function of the" 
chants in the ACTICS-II^CRIHriCS pver tii^.. 

threshold va^ to vhich EEIZPA-I ig oon^axed' to 
Jiidge th^i emoted payoff of deteiaining further 
obsefvati(^4 ^ - 

threshoiJ that detenaines yniMrmTTn p^off of 
obsenrationa niade* . * . - 

threshold that detennines maxiisum ccet of^jtepairs 
made* ' : 



Procedure invoc ation semantics 
arc-label (from Figure 5) 



a 

-b 

c 
d 

e: 



: If'ERROE? = TRUE ' . ' ' 

thfere exists at ieasrfe (jrie- ol^servation with axi expected p&yoff 
greater than 0. ■ " 

if M/KE-:ESP-0BgEB7^CH was called. 
ifJEBBOR?*= FALSE and ISnEA.I > L " • 
if the cost of ^repairing the parts denoted by the totil 
LOC^CS-HXPOT^SIS is less than R. ■ . 
if the cost of repairing the part Jssiaaed to 'have been 
, recognized ^8 the location of tte'^ault is less than a. ' 
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teleological structure is implicit In the sen^ntiq testa €dr cpnditionar 
procedural '■calls. The. tests involve global registers and data' ' - 
structures thtt'are accessed and modified by procedur|8 throughout the 
^ network. Thus, the invocation of co^fiitional procedures and , in fact, 
the- a.ction8 of both conditional'- and uhcondfitional proc^d'ures 'depends' not 
only on the actions of th? calling proceduri but on any procedures ifeat 
have prev^o^ly BodifieiTthr registers and dat^i structures. He a2: 
-.thi/point to emphasize that while the procedures contained in a -general 
troublesfaooting^ebugging strategy may seem obvious, the relatioijships 
between them are not andnay therefore cause the greatest difficulty tn 
understanding and inducing how to apply the strategy by observing it in 
action « ' • ' . * 

♦ We will now proceed- to elaborafce the fiiodel, describing the . " 

calUag semantics Vd f unction of each procedure and indipatlo? the 

- : -differein: types of knowledge required 'and how they become available to' 

.V. ' ■ - • * 

n/ the problem' solver. 

, TEST. Every time a plan is executed or a system is^activated„ 
it , is implicitly being For instance, whenever you switch on^a^ 

. light, you are testing -^t and^the cigpuit of which it is a part. If thf. 
.light fails^o-go on", then debugging is initiated. More clearly, a" 
techni#an engaged in routine maintenance 'co^ciously tests a system to- 
. ^ see If he can gather data which may^^sauee him to reject the hypothesis 
that the system is fully Vrational. Thus, the top-level p^ocedure^ 
the model is TEST. The model Is. aii^^s started at TEST. At that point, 
•a register ERROR? is FALSE, *indicat'^ng In assumption that tl)ere iWno 
error in the system being tested. This is also reflected by the initial 
value 'ot. a data structore ACTION-^ESGRIPTiQN which is NULL. TEST Is' 
Q also palled, by REPAIR. ' * . ' " y 
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. TEST Invokes .CHARACTERIZE unconditionally. It subsequently ' 
calls DE3UG only if ERROR? is TRUE ut)on return from Cl^CTERIZE. 
^ ' CHARACT-ERIZE. The function of CHARACTERIZE is to Collect data 

that allow modification of the ACTION-DESCRIPTION. If it adds a clause 
. to the- ACTION-DESCRIPTION that describe a discrepancy between observed 
and .normal actions, then it sets ERROR? to TRUE if it was previously 
' FALSE* This jrorresponds to detecting a bug. during testing* , 
CHARACTERIZE does its work via three subprocedures, 
DETERMINE-QBSERVATIONS,, MAKE-BEST-OBSERVATION, and , 
, .MODIFY-ACTION-OESCRIPIION. The call to MAKE-BEST-OBSERVATION is 

coaditional on whether there is a potential observation whose payof f Jfa 
function of its post and expected information r#g|)^exceeds a minimum 

V threshold, which we will denote 0. This means t^an observation is 
■not made if it is too expensj.ve or if it is not expected to alter the 

ACT^-DESCR^TION significantly. " The initial value^of 0 is set bj TEST 
and depends on the expected cost of a subsequent system failure if a bug 
18 not detected and repaired. CHARACTERIZE also may call itself 
conditionally, if ERROR? is FAL§5 and, a register" DELTA-I, which reflects 
the rate of inforsatipn change in the ACTiON-DESCRIPTION is above a 
threshold I.' This means' that when CHARACTERIZE^s called by TEST, 

V - either, Initially or after* a repair, obse^ations sill be .made as-long as 

the ACTION^ESCRIPTION changes by the addition of propositiogs asserting 
that observed actitfns are normal or by the' deletion of pro'positions 
asserting discr^cies noted, previously between observed and normal 

♦ I 

actiKjns. In general, this implies that chatacterization during testing 
continues until there ai^ no more potentikl observations whose' payoff 
exceeds 0. tlius, testing does not hfecesgarily continue until \he 



debugger is logically cer'tain'the system is e^ror-free. bu^ onl/unttl. 
his confidence leads hip to believe that further obse^atiSns have a 
higher cost than failure to detect a possible error ^ould have. 

DETERMIME-OBSERVATTnMS, ih_e first procedure called by 
CHARAnTER;ZE is DETERMINE-OBSERVATIONS , vfaich identified a set of . 
pptential observations." The observations are determined with respect L 
the current LOCATION-HYPOTHESIS, a data structure describing a' i' ' 
subhierarchy of the -plan/system which is initialized to the entire 
hierarchy and modified subsequently by the procedure ^ ' ' 

MODIFY-LOCATION-HYPOTHESIS by deduction Involving the 
ACnOii-DESdklPTIQN. Th* LOCATION-HYPOTHESIS represents the part of, the 
plan/systen. to ^ich a detected bufe" has been logically .isolated or 
conversely may be viewed as^t^ part of the plan/systeo which is not 
known to be bug-frerf-grven a^'prior observations. Each observation 
'identified .by DETERHINE-OBSERVATldN^ iias^ a potential eff'ect on the 
ACTION-DESCRIPTION Ohich can further reduce the extent of the 
^Ian/system denoted by the LOCATION-HYPOTHESI? , 

. Observations may be experimental, involWng manipulations of the 
plan/systems parameters (if any). For example, they may require an . 
electronics technician to change the external control settings of a i 
device, pr a programmer to alter the data input to ^program. In 'some j| 
domains,- like management consulting, no experiments are possible and 
observations must be "naturalistic.",, 

. PETERMlHE-OBSmVATIONS accesses the debugger's knti^^ledge of the 
plan/system's. functional hierarchy and its teleological structure in ' ' 
order to identify points where informative obsen^ations can be made. In 
some contexts (e.g.. el^tronics trouble8ho'tfting>; there-pray be external 



sources of that inforaation (technical data). Otherwise the hierarchy 
sAsimt bfe built up from the lowest level using knowledge about primitive 
subparts an* the laws tha? describe theif interrelationships. Knowledge 
about higherroraer sul>part8 derived in this way may be sjtored in memory 
in a "library" which may allow the debugger to recognize that subpart if 
'it appears in subsequent episodes. ' '/ 

' HAKE-BESt-OB S'ERVATICN . MAXE-BEST-OBSERVATION is second 
procedure called by^ CHARACTERIZE. As riotecf in the discussion of 
CHARACTERIZE, tts call is p/mditional^n Chere being at least one 
potential observation with an expected payoff exceeding 0. 
MAKE-BEST-OBSERVATION performs ^e observation with the highest payoff * 
as determined from, Its cost and its potential ' for affecting the ,i 
■ ACTION-DESCRIPTION. An observation expected to" return a large amount L 
information iqay be passed ov«| for a less product ivfe one if the latter 's 
cos't is much lower^ " 

MAKE.-BEST-OBSERVATION accuses tYie debugger's knowledge of h6w 

to BSke observations (e.g.,^ use of measurement equipment) and of their 

^ ^ _ ' ■> 

fejfpected cost.- Most kriowledg'e of these" costs is probably acquired 

thi^ough experteace and'taay bfe scored in the saiie library ,as the 

knowledge used to rec^iz^ ti5^her^rder subparts. UTat library 4ay 

also contain t6e knowled^ of likely optcom'es of observationa- used to 

' •* ' - ' ' ' ' 

estimate "the effect of ^n okservgcion on the ACTiaH-OESCRIPTION, This 

' I . * • ^ ' , 
latter knowledge- suppl8ti|rt# information about the possible outcomes 

deduced from Jcnowledge of th^functidnal hierarchy of -the plan/systea. 

MODIFY-ACTtO N-OESCRyTTOH . . This procedure modifies the 

ACTION^Etomoti, according 'to the observed actions and is called only 

lf^MAKE-BEST-OSStfeVATnl|^as called.' The modification involves adding a 
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proposition to the description noting either a nonsal action or a 
discrepancy froa a^noraal action. , In testing subsequent to a repair, it 
may also involve deletiog or sodifying a proposition already' in the 
description, X^eneration of the proposition requires access to kaowledgp 
for deducing the nonial^ctions of subparts and structures of subparts, 

. DEBUG. DEBUG is the controlling procedure once an error has 
feeen detected, 'it is called by .TEST if CHASACTERIZE'has returned with 
ERROR? equal to TRUE. It calls the procedures RECOGHIZE-BDG, • ' 

MOd'ifY-LOCATION-HYPOTHESIS, REPAIR, (kaAJTERIzr, and itself- ihe call 
to REPAIR is conditional. Further details about DEBUG will be given 
following the descriptioa of Tts subprocedures. J * 

RECOQHIZE-BUG. RECOGKIZE^UG is a powerful procedure in that it 
can ra-dically alter the overall strategy of logically localizing a bug 
at i^rogresslvely lower levels of a plan/systes's fuactioMl hierarchy. 
It accesses the ACTION-DESCRIPTIOH ^nd Batches it against a knowledge 
library of bugs and associated ACTIOH-OESCRltT:iO»S encountered in past 
ep^isodes with identical or sisilar plan/systeas {idiosyncratic 
experiential knowledge). U a sufficient natch is obtained to a known 



bug and the cost of repairing that bug is is less t|an a threshold S, 
then RECOGHIZE-BUG issediately calls REPAIR. If the "gxist of t^e-fepat 
is too high to be attempted at that tlae {R increases as a function of 
the number of tiaes DEBUG ba^ been called), then the old 
LdCAIIOH-flYPOTHESIS is saved and a new LOCATIOH-HYPOTHKIS is set to be 
^ the level of the hierarchy at which the,, subpart containing the 

•recognized bug is dftfined.^ Ibis has the effecu of focusing subsequent 
characterization on a "-suspect" subpart. For txampU, when a aecbanlc 
Eirst exasdries a car with an uneven idle, i:he ACTIOH-^ESCSIPnOH is 
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■■uneven\lle" and the ioitial LOfcAIIOH-flYPOTHESlS Includes the entire 

r 

ignition and fuel systems. If he has knoviedge that "uneven idle"' is 
fr^uently due/to dii^t in a carburetor passage, and is faaililft with the 
'trick" of jatring the dirt loose by striking the carbtiretor on the 
outside, then he say. isEi^lateiy try that repair* If he is not familiar 
with that inexpensive repair (or if he is and it doesn't sees vork) 
and is not yet ready to disassemble the carburetor or use a chesical 
cie^eF, then he can sef the LOCAIiOH-flYPOTHESIS to be "fufel systes" so 
that he can cake further observations vhieb viil indicate whether or not 
there is sane problen in the carburetor. If the pggbleg^ ^gically 
localized to the carburetor, then an appropriate repair will be sade 
vith the savings, of not having isade unnece^ary observations to exclude 
the ignition systea. !k>wever, if one of these observations on the fuel 
systes should sake the LOCATIOH-HypOrHESIS logically inconsistent vith 
the ACTIOH-DESCRIPTIDH {as detected by J©DIFT^X)CAXIOfi-^PDIH£SIS), then 
the previous LOCATION-HYPOTBESIS mst be restored and ^ified. Ihus, 
for exasple, if further, observatlons^rove conclusively that there is m) 
fault in ihe carburetor!" then the LOCAHOH-HYPOTHESIS containing the 
ignition systes and entire fuel systesg is restored and the 
problem-solving process continued fros that point. 

HODIFY-LOCAIIOH-flYPOTHESIS . Ihls procedure accesses the • 
ACX^N-DESCRIPTIOH ami using knwl^ge of the plan/sys tea's teltology 
deduces whether any of the subparts in the LOCATIOH-HTPOXHESIS logicaUy 
can be excluded as candi^tes for containing the bug. Thl^ Is 
'illustrated by ourearlier example of troubleshooting when your bedside 
lai^ falls to light. Initially the LOCAXIOH-fi¥POTH£SIS inclttdes all the 
ele^eots of the circuit. When the observation is sade that the electric 
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dock is still working, the -ACTIOH-OESCEIPTIOH becoaes "light 

« 

inoperable, portent available at wall outlet." 

lS)Dm-L0CAIIOK^YP0Ta£SIS deduces fro. this that the fault canhot be ia 
the external power supply, the fuse, or the interseJiate wiring and 
Hxxiifies the LOCATIOH-HTPOJHESIS accordingly. 

When the LOCATIOH-HtPOTHESIS is reduced to a.sinfele subpart at a 
level of the fimctionai hierarchy, it is reset to contain the subparts 
in the level iaaediately below that subpart. For instance, in * 
. troubleshooting a circuit, Ji the LOCAIIOK-HYPOTHESIS is reduced to 
•voltage regulator", it is reset to the level of the hierarchy 

r 

c«K:?risiQg the issediate subpjrts of the voltage regulator. Thus, the ' 
' bug is localized to progressively sij^ler {and structurally ssaller) 
. parts -of the pian/systes. / 

As oot^ in the discussion of EEdDC^IZE-BDG, if- 
HODlFY-LOCATI0H-4iYP0TH£SIS should deduce th?t the ACTIOK-DESCRIPTIOH is 
inconsistent wiph a bug anywbere in the parts of the plan/sy6t,ea denoted 
by the LOCATIOS-BYPOTHESIS, thefi a prior call to RECOCSIZE-BOG produced 
a false recognition and the LOCATIOH-HYPOTHESIS prior to that caU is " 
restored. , , 

* REPAIR. If the cost of repairing {replacing or aodlffing) the 

8ufapart{s) denied by the LOCATIOB-HYPOTHESIS is less than the repair 
cost threshold R, ,then DEBDG calls REPAIR. REPAIR accesses the 
debugger's knowledge how the subpart (s) are denned and iapleaented to 
function as intended. For an electronics technician this say be, 
* soaethlng as basic as how a transj^tor is supposed to be conaected and 
^ for 3 prograa»er bow to write a forskt stateaent. On the bther hand, a 
prograsaer-aay rewrite an entire sorting procedure if he deteraiaes that 
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there is a bug in the existing one^ and belies it is tsore efficient to 
rewrite than to try to localize the bu^ Sutthei* REPAIR also accesses • 
knowledge about specialized "tools" '^Llke sblderlng irons or cooputer 
file editors needed to acc(^lish repaij?/^ in different donains. 

REPAIR sets ERROR! to FALSE,<^ calls TEST. If the repair 
corrects the fault then thatj^*! to TEST will eventually call "STOP, 
tetatoacing ^he problen-solving process. If the repair is incorrect, 
the call to TEST will eventually invoke DEBUG again. , 

' Continuing DEBUG . If the cost of calling REPAIR exceeds R, then 
DEBUG -calls CHARACTERIZE. and then .itself . Upon the return fros 
CHASACIERUE, the ACTIOS-OESCRIPTIOH will have been updated* if an 
observation was sade. If one was not, then DEBOG aodi^iefe both R and 0. 
It increases R, so that there 'is a chance that REPAIR can be- called even 
though the LOCAaION-HYPOTHESIS cannot change because the 
DESCRIPTION-HYPOTHESIS is not sodified. It decreases 0, so that if 
.^^^^ ^^}^^ cannot be called there is a chance that an observation will 
be eade oa the next call to CHARACTERIZE. Thus, when the process gets 
stysied, it frees, itself either by sakin^ aore expensive repairs than - ' 
usual or by saking observations that are sore expensive or less 
informative than usual, — 

^^^^^ cogsects on the isodel . The strategy ire Imve outline 
here is a co^etcnce rather than a perforaance sodel. Deficieo^ics in. 
any of the knowledge required say cause it to fail* In particular, the 
f knowledge of each level of the plan/systCT's functional hierarchy and 
its teleological structure is crucial for aodifying the 
ACTIOH-DESCRIFTIOH ^nd the LOOO/IOH-HYPOTHESIS. Note that it is not 
necessary to know the hierarchy froa top-to-bottoa but Instead only dotm 
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to the level at which one is wUling to pay for a repair.* Thus, in 
working on a circuit one may understand (froa technical data) the ' 
functioning of the voltage regulator with respect" "to oither subcircuits 

.at that levei of analysis, but not understand the internal stmcture of 
the voltagi regulator. The available knowjedge'is sufficient for 
localizing a fault to the voltage regulator and this say be adequate if 
one is'^wUling to replace chat entire subcircult (knowing that only one 
primitive component nay have failed). 

The only explicit error recovery eechaniso in the sodel is for 
false recognitions by RECOQilZE-BUG that cause an inappropriate jump 'to 
a lower level of the hierarchy. The oodel backtracks froa these errors 

by saving and restoring earlier copies of the LOCATIOH-HYPOTH^IS. 

Thus, these errors increase costsT but will not directly lead the 

process to co^lete failure in the way other knowledge deficiencies may. 

Explaining the expertise of expert debuggers .' 

Given that a general strategy and different types of 
donain-specific knowledge underlie troubleshooting/debugging behavior in 
the ways suggested by oyr oodel, we can ask-about the contribution each 
nakes to expert performance. Is the expert an expert because ^ 
'develops a superior gea^ral strategy and adheres to it religiously? ilr, 
does his expertise stes aore froa his extensive knowledge o^ the jfjroblea 
domain, including the fundaaental declarative knowledge, specialized 
procedures for saking observations and repairs^ and idiosyncratic 
libraries of inf oreation about important recurring 'high-level subsysteas 
and about the bugs, frequently associated with observed patterns 'of ^ 
symptoEs? The intro"^ectioQS of the. expert in the dialog in Figure 2 
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are coasistent vrl.th tjie^atter explanatioi). He ^attributes hi^ easy and 

efficient solution of a problea to his "familiarity" with the fact that • ^ 

# 

^ an observed syiap ton is (alaost) always assoclat|d with a partlciilar 
% fault, although he cannot articulate how he was able to access that 

fact* . 

In ter^ of our oodel for a debugging strategy, the expert in ^ - 
the dialog -^hlev^ a solution by calling his REC0GNI2E-BUG procedure, ' 
bypassing some of the progressive top-down localization and 
characterization which slowly converge on a fault by deductive 
reasoning. He suppleiaent^d his deduction with Identification . 
Localization by Identification is also exe@plif ied-by^ the mechanic who 
first Jars the c^rtTuretor to afcteopt to remedy & uneven idle, ^ese 
'examples illustrate how by using a library, of kiiowledge acquired through 
experience, the expert can choose to focus at a *low level o£ 
plan/systegis functional hierarchy without deducing the location of the 
bug froQ observations nade at higher levels* However, since the 

Inforsation in the library nay sooetlaes be applied in inappropriate 

♦ 

contexts (false matches to ACTIOH^ESCRIPTIOKS),^ the expert, mist 
backtrack and integrate observations aade at several levels in the 
systeia such laore frequently than required when localization is strictly 
top-down and decfuctive.. Failures in backtracking can reducfe efficiency 
by causing the expert relying on identification to sake r^undant ' ^ 
» observaltions. * " * I 

This explanation of debugging expertise seeas to be consistent 



with data we collected froa five prograisaers with different degrees of 
* 5 

experience* They ranged fros one with a easters degree^in coiq>uter 



. Whe/we say "prograsaer" we do not necessarily aean an 
individual who was trained' and works specifically as a progr^aaer* For 

O * 42 . 
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science and several years of advanced programmltig experience to a total 
novice who had no formal instruction in programing and only a few weeks 
of self-instruction. Within this range were students and professionals 
with^froza one to several years of instruction and practical prograa^lng 
experience* 

We asked these programers to debug a short BASIC progras and* to ■ 

* 

write a comi^ntary on their reasoning as they worked. They had access 
to a BASIC interpreter for running and modifying the "bugged program^ • 
rney were provided with the prog^^aa description, listing, and' sample' 
input-output shown in Figure 6, It is a sorting program designed to 
interact with a user at a terminal. It accepts numbelrs one at a tise, 

<r 

acknowledging each by printing- "BOK AFPETIT", until the user .types A 
zero signifying the end of the list. The list sorted in ascending order 
is then printed. The program was written by a member of the research 
team and is deliberately obscure so that it would riot be cotapletely 
trivial to experienced progracmjers despite its brevity, A ©ore elegant 
solution is possible using fewer variables and less complicated 
parameters fj>r FOR,,HEXI loops. However, the program shown is correct 
except for a single character. Line 100 should beX«C-J+I instead 
of X « C - J + J,, Hote that such an error could be the re8u|.t of a 
simple reading or typing error in entering the program, rather than a 
design error. However, it is not the type of error that can be detected 
by the parser or' runtime system o£ a prograsaing environment j it is a 
logical error ^hat causes the program to produce inc<^rect results when 
it Is executed, as can be seen in Figure 6, In fact, the effects of the 



our purposes a programmer Is anyone 'who writes programs incident to bis 
job or (his activities a student), , ^ 
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Dw-icription 

This program inputs numbers (up to 100 numbers). 
'• sorts each number into ascejidmg order as it is input/ 

and (prints the ordere-d inputs when a key value *of zero is input. 

10 DIM N(IOO) . * 

20 C = i 

30 INPUT N(0) ' 

4Q IF N(C) 0 THEN 180 

50 FOR 1 = 1 -TO C 

60 IF fcl(e) N(I) THEN 140. 

80 D = I . . ' , ' ' 

/90 FOR J = D TO C, 
100 X = C - J f 1 

110 N(X + 1) = N(X) • 
i:?0 NEXT j' 
^ 130 N( I) - N(C + 1 ) ' 

140 NExr I 

150 PRINT "nON APPETIT? J 
160 C * C + 1 * 
170 GOTO 30 

, laO FOR Q=lTOCTi " ^ 

190 PRINT N(Q) 
200 NEXT Q 
210 END 



Sample input/ootput: 
»RUNi 

EXECUTION OF YOUR PROGRAM 





. 77 


BON 


APPETIT 




: ID 


saN 


APPETIT 




: 34 


BON 


APPETIT 




: 10 


BON 


APPETIT . 




; 88 


BON 


APPETIT 




:0 



10 
1!> 

0 

77 
88 



r 
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EXECUTION COMPLETED AT LINE 210 

* 

Figure 6. DebuggiDg problem given to five prograamers of vaiying 
experience. 

■ ' - 

kk 
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bug depends on the input and if theuaer ?ntertf a list in perfect 
reverse order (e.g., 81, 54, 33, 12), then the prograa correctly sorts 
it. A cpB5>lete ACTION-DESCRIPTIOH of the bugged prSgrai^at its top 
level is that if an incoming number belongs at the beginning or the end 
^of the. existing list it is appended correctly, but if it needs to-be 
inserted between two previous values, it is lost and replaced by a ze^o. 

All of the five prograaiaers who participated were .able to debug', 
the prograa, although the atsount of tige they required varied froa less 
than an hour for the aost expert to several days (a few hours feach day) 
for the novice. Also, they varied in' how aoch of the prograa -code they, 
nodified in order to elialnate the bug. Figures' 7a, 7b, and 7c are 
segments of the written consentaries generated bytKe expert, an 
interned iate prograaaer, and the novice., respectively.' The concepts the 
expert uses to describe the prograa reflect his specializi knowledge of 
sorting algorithms, which he usefi to identify functional subs^gaents of 
the prograa. ' He iaaediate tried to identify bugs at a low level of 
prograa structure - in stor^e -allocation and in li^cr^enUng counter 
variables. Although he exaaln^d the sample output at an early point, he 
does not articulate an ACTIOH-OESCRIPTIOK. The stiructure of the episode 
reflects repeated atteapts to use specialized knowledge to predict and 
search for likely types of bugs: an eaphasis on recognition as opposed 
to localization. His repair, not shown in the coasaentary, was to ^■ 
rewrite the nested sorting loops in a sore .straightforward torn, 
eliainating the unnecessary variables. Thus, he aade a judgeaent that 
it would be aore efficient to substitute a block of code, than to -try to 
Isolate the"Hug furl|ier and then make a isore ainfaal i»diflca|ipn. 

. Figure 7b is the coaaentary of a progtaaaer vith several y^s 
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• J/' 

f 

Read description. 

Check HEM statessent. 
Look art output* 
^ Bon Eqppetlt? 
C pointa to next einpty vordL m H 



C Sc. I 

When input = 0, then ISo? print H(l) to"H((M) or stop. Logk for C ^ 
Pound on l60.- . 

Sort tates place between '50 etod lltO. If H > all of the B"^*s, nothing is done. 
Get' to 80 if H < H^. 

The J loc^ (9cjl20) is sxipposid to move gll o'f the entries between H, , and K 



np one. 



It does Bcxne sort of inversion. I think this is unnecessary, and also^the variables 
"D X. I as goilig to tiy it out on the teznin^. ^ / 

Attached is the^ program I typed in, before I tried tc run it. (Hote that It 
differs froctt ecr^Oibled notes. I didn't look at the notes wMlfi I vas typing . 



it in.) 

I tii^ed sorting tbs nmabeit 

9, k, 6, 1, 7, 13 
and it voxked. 



> Figure 7a. Debugging cccinen^iiy on porting program by a foimally 

t ' trained, highly ezperieif^ progrsBBier, 

1 ■ 



k6 



ERIC 



54 



Think about what it should ^o. Visualize nxanber coming ia andofinding its 
vay to Jthfe top of list* 

* 

Hot^ that 0 is in. the vrong^ place and 3^ is missing — in .feet," 0 is in the 
3^ place. . 1^ 

3** is the 3rd input (check first to see if- it's first or last — no)» 
Nov go to program. . * r ^ 

Sincel 0 was vrong^ chedc the branch^to 130 . Don*t see anjrOiing vieid .there 

output loop looks CK» " . _^ ' 

Look at sort loop (50-lUO 1 guess), vhich skips if culrent (n^ inputy Is geq.^ 

Inner loop looks complicat4a||^act, bizarre. Have to'puz^ it through 
purpose is to shove everybpdj^Pfi to insert neV guy. I knev that b^cauge It*? 
a loop (should have knowp it anyway). * * * ' / 

lOO puzzling next-one Ih the list. Pus,h it erne lower dtwn. Looks bad 
H(x-fl) -^N(x) will Idse the prevL^a value of.R(x'fl>.* (hypothesis): Ofest vith 
given inputs by 'tracing valuM|^^™^2ything. ^ot walking — I'm not, getting 
anything pushed anywhere exc€^^pK^crf si^ the 15*is moving higher instead 
-^rf the 77. Try again. . ' ' ' * ^ ' ' 

Rote: D is ^unnecessary since tySSesn ^ tr change inside the J loop — I vouidn^t 
be destroy*a if 9 0 for J = I to c/ TMnk ^bout that:, is this strategy 
reasonable — ye^-JTT^ve a place where fife ^curreirb "Input is lower than this 
element, so; I have to change everybody frcm here on up. So J I t^-'C-iB fitife. 

EUt I Immedia'tely get N(3) ^ 15 which is dumb* (Think about it^^Jlfer BIP thinks 
X,= C-Jfl gives. 2 or if 0, then 'array efror would have occurredTso imist be '2.) 

Seems that N(2) =. 77 N(3) « 15 vhich is backwards. . Keep gcdng. C changes 
to* 3 but t^^t -should have zapped the 15 in H(3).^ Ko — missed I30 wktlSr stuffs 
15 back^to 111 Seems that I get 2, H(2) ^CCTipares. KO: became H(Ci) is 
no longer the cui^nt l|put. 

Looks like I -jtlpt lost the 77 but that*6 impossible, so have to try again 
with 3^' input./ * ^ ' ^ . ^ ^ 

V. . ■ ^ 

Wait:. Strategy makes sense put new one upJiigh, move the others up one at' 
a time. l^lW gets the ^If, 'so N?3) is wsvailable for 77 to moVe i;o. mxie. of 
X should be 2^ to make that mov^ X ^ C-J-^kL is 3. 

» * • ^ ■* 

•HO. The ti,rtt time was wrong. Back up. to 90 again with 1 = 2 (i.e.-, cotnparing 
3i» with 77Ti ' . , * 

" " - . H . ■ ' ^ ' 

- Figure 7b. Ext^ipt^rt^ debugging commentaiy on sorting progim 
^ < • / fey a progrananer^ of intermediate ejcperience'.' v 

A • * ■ . j\- ' - 



No, wait a secpnd, J guess it*s O.K. for it to print 0; it should 
just" print it first • Maybe that's where the problem is. Even 
though I still haven* t^ figured out how the program is supposed 
to work, l»ll check out 'the' parts that deal with ^switching froift 
the ordering proSedure itself to printing the final output^ and 
see' what i can,f-ind. ' ' 

Of course! Zerjf/^ is going^to^be the last valxxe entered, "so 
it will have the highest numbered subscript and get printed, last. 
That's certainly one of the^ jirob^ems with this program, although 
, it may *not' be the only one. 'l'll,.hava to figtfre out how the 
whole thing is supposed to work. But I have 4 hur^ch that if 
line 40 ("IF N(C)=0 THEN 180") gets moved down between lines 160 
and 17^ (the end ot the main subscript-rea-ssigning loop) then 
the program, will work the ^ay it should. ^ fhis move should assign 
0 the^^qwest subscript before telling th6 .machine to print out, 
assuming the rest of the program was written correctly. Let me " 



check that, out". 



/, 



'160 'C?=C-fl ^ ' - • — so that's how the subscript 

gets incremented*^ 



(Break — overnight) 



' * t ;just realized that although^ 0 was initially assigned the 
highest subscript (I think), it -was notJ printed last when the 
numbers wer^ printed, out in ^ubscri^t order (lih^;5 18D to 200}* 
"This means that the subscripts of some of the other numbers, 
(namely 77 and 88) Vere higher than the subscript for 0 by the 
t4me the printing was done^* I'jn gping "bo try to run through 
the program mentally/ feeding- in ^ the ntimbers-^used in the run 
shown here, to see what this program, is'doing. 

when you get to INPUT 3>t, 

• ' N{1)=15 'N(2)«77 

N(l) no longer *has a value^(2y=15 l^-f3) =77*-3r4 is lost here 

•N(2) nb ^ong^r has ^ value^(i)=15 N(4)^71 ^ 

/^though I still don't grasp it completely, t)ie , strategy ' 
of this program seems to be .to , reorder the/ subs crip t's^f th3 
input jiu;nbers by comparing eachx new ntimber one at a timfe with 
eao^df the^^numbers Which ^lave already b^en-^ped in and: 

— If the newf number N(C) is lesjs than the nyjislb^i^^z^^xs being 

compared to, M(I>, this increment- th^ subscript of each of the 

numbers thatlRre greater than or egiM^to N(I) ^nd 'gdve* the new 

number 'the old subscript of N(I). , . ' / 

* * • ' ' 

--If the new number N(C) is greater than the old .number N(I) to 
which it is being|coinpared, leave the ^subscripts alone and 
compare H(C) w4.th the next -old number .\ '^'l . ' • 

Fifeure 7c. Excerpt from debugging -conimentary on sorting program 
J , "10'^ no'vice programmer* , . 



of practical experience, but little formal instruction.'' There Is nsore 
evidence of a general debugging strategy and less use of specific 
knowledge_^ about sorting than in the commentary in Figure 7a. Before 
looking-. a|t' the structure of the program, the programmer tries to. 
describe the bag's symptoms in the program oMtput— to develop an 
initial, fop-level ACTION-DESCBIPTION. ' The observation "that 0 is in 
the wrong place" {she incorrectly awHimes that the 0 In the otitpit Is 
the 0 the used typed to end his input) leads her Co locate a"^ 
characterize the segment of the program d^ned to st^op'the input cycle 
when a 0 is typed by the user {she sets mlx)CATION-HYPOTHES IS to that 
segment). When she {mentally) tests that'se^ment anck- observes' no' ' ' 
evidence of a bug. she focuses on th'e nested loops that sort the array, 
(resets the LOCATION-HYPOTHESIS to the top-level) ) Her reasoning 'in 4 
using the zero in ^he output, represents a.call"from ^UG to * > 
RECOCNIZE-BUG in which an incorrect ACTIOS^ESCRIPTIOH was matched to 
information in a library of bugs and their manifestations. 

The programmer's experience allows her to ^udge that some of the 
code in the sorting loop is "bizarre", and thus' a likely location of the 
bug. She characterizes, that code b^ mentally tracing execution "^ami ' 
observing how ^ao ac^^f (as opposed to abstract) set of numbers are ' 
mbved within tbe-^rray. She has some dlfMcultf in generating an ^ ' 
ACTION-DESCRIPTION from her observations and therefor* tries to "apply 
some knowledge she does have about" sorting' when she decides that line 
110, N<X + 1) « H(X), looks suspicious (another call to RECOGHIZE-BUG). 
In some sorting algorithms, such a transfer 'o§- values might lose^e 
contents of an array location, but in this casd the lloe is correct. 
(One of the ofher intermediate prograaaers also "suspected* this line^ 
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Thus, she.has^ to kac^track 'f roa this atteapted identification of the 
error md resunes her characterization of'^g segment of the" ne^t^S loops. 
Eventually, she did identify and repair Ihe bug. Like 'the expert, she 
tried to^use specific experiential knowledge to shortcut a top-down 
analysis via identification, but did not have the knowledge ^ded to 
succeed on, ?ha.t basis. (Slje night have solve(^tfa^^obles acre quickly, 
but unlike tM- sore expert programaer, chose to \ocalize the bug within 
the stating loops rather than rewrite then compile tely.*) ' 

Figure 7c is one of six pages qf conaentary generated by .the 
novice progranmer {who later displayed better than average pr«>gratoing 
skills for his degree of experience) . He wo^ked-T,n Che]" prograa .in four 
sepa'rate sessions and eventually 'did debug it. " His coaaentary revea'lsV 
how his uafaaillarit| with the fundamentals of the BASIC progran^ing 
language and of prograa organization laade it an effort for his 60 
perceive the prograa at a high level. He begins by spending 
considerable effort characterizing the code line-by-line vith no good 
idea of &hat he is looking for, since he f^ils to j^eoer^te an 
ACTION-l)ESCRimON beforehand. In fact, he did not report looking at 
the, input-output data befo^re the second session!. Prior to his 
successful solution he attebpted several "ijratibnal" ainor repairs 
based im his. misunderstanding of single lines of code and their function 
in the prograa. Like the aost ^pert prograaser. he searched for bugs 
by ^aaining the, code, but Unlike -the. expert he had no basis for asking 
rational predictions for what the bug eight be. 

' In general, these* data are consistebt with the view that' the 
ebcpert debugger is an 6xpe.rt~ that he solves problem with ainimal ' 
expense— because he has a ^reat deal of experiential knowledge that 



allows hia frequently to follow cost-saving 'alternative pathways wi^tljiif 
a general debuggingr'strategy , es represented in our aodel by the ^ 
procedure RECOGNIZE-BUG* It is not seeo necessary to postulate tha€ he 
has g^eneral strategy superior to that "of sose^ae less skilled 
debuggers in order to explain his expertise. Instead, he siBply seeM 
better able to ^exploit the benefits of an identification substrategy 
which even novices try to use. 

* Weaknesses in the debuRj^ing of inexperienced progr^^^^rg 

The connentary in Figure 7c shows that an inexperienced 
prograncser can have considerable difficulty with a debugging probles 
because of the effort required to understand how the progras is supposed 
to accc^iish its intended 'function. "Of course /programers isost often 
encounter their debugging probleos in prograss which they themselves 
designed and icpletaented, asiS thus can understand. Bovver as we noted 
earliej, prpgranners sonetlcses knowingly ij^lement and run programs that 
are incorrect, finding it isore efficient to develop correct code by 
debugging, than to derive the correct code initially by logical 
analysis. In these cases, prpbletss in debugging^ can arise becatise of 
difficulties in knowing how-to design code for repairs, rather than in 
locating the bug. Sonetises,- presumed understanding of soae code can 
actually impede progracners' debugging of their own program^. If they 
write code they are certain is' correct anS isanage ,to insert a bug in it, 
then (1) that code is the last place they will look for tBe bug, despite 
observations that ssight indicate tli^t it is a likely location, and <2) 
when they do look at the code, «|y say siss the bug, because they see 
what they intended the code to do and not what It actually does* Thixs, 
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a prograaaer debugging his own prograo liay lose sose objectivity:, jwbile 
one debugging another's prograa ©ay have fundasental problem 
uoderstandlng how the prograa i^ supposed co work. There are soae 
prograsoers in real contexts who are faced with the problems of 
debugging programs written by soneone else: for ^aspTe, consultants and 
cenbers of teaas working together on a large proje«. Tney lose the ' 
advantage a designer's knowledge- of his program, but by the ^ase token 
are less prone to "blindness." Taev nay face situatioo^^ere they have 
difficulties debugging a pfogran because they dori't have the knowledge 
needed to understand it, rather than because they have inadequate 
debugging strategies* In other troubleshooting /debugging domains, like 
electronics and oechanics, technicians routinely face proble!^ with 
devices unfanlllar to then* In these situations, they mist turn to 
technical da^ for the devices or be able to^synthesize^ the device's 
structure fron the botton up, if they are to effect a repair. 

We have proposed that expert debuggers have general, top-down 
debugging strategies^ but that their expertise is defined by their 
cental libraries of docain- and probleis-speclf ic knowledge gained 
through their experiences. Inexperiences progracaers obviously lack 
cospreheosive libraries. But is this the sole source of their 
difficulties, or are their generai debugging strategies also deficient 
so that they do not aake the eost effective use of the specific 
•knowledge they do have. * This is an iiq>ortant question froa the 
viewpoint of instruction, since it would be sore feasible to try to 
teach a well-defined general strategy, than a large, ill-defined corpus 
of specific knowledge. The coaaentary ot the novice prograaaer 
debugging on the sorting progras does seen to reflect a stpategy less 
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efficient than those we found in the cotaentaries of sore. experienced 
prograssers vfao also had difficulties with the problem. However, the 
knowledge required just to understand that program was so far beyond the 
experience of the novice- that it could have been the case that he bad a 
good strategy available but had trouble eietiutiag it: 

lo an attcEpt to detensine whether inexperienced prograasers 
nave difficulties debugging b^ause they' lack an effective general 
debugging strategy, we examined |irogras=ing data collected fro= students 
learning to prograc. -Tbe data originated fros-three groups who had 
participated in the BASIC Instructional Program (MP) 1975, a C&l systea 
for teaching introductory BASIC progracaing to people with no prior 
(forputer experience. In all there wer4 data fron 100 college students, 
vbo wrote on the order of 40 short BASIC prograes each during 10-1 i 
nours of terminal "tiise in 3IP. Tbe original use of the data had been in 
evaluating BIP's effectiveness for teaching progras-iing and in exaalning 
the. way students used sooe of BIP's subsystems for writing and drugging 
their prograns. 

rae data are quite casprehcnsivs records students' 
interactions at the tensinal, which we will call chrotologies here. The 
Inforcation contained in the chronologies for the three different groups 
varies sooewhat, since analyses of the earlier versions had suggested 
Icprovesents in fomat and Content. . For instance, ihe first and second- 
groups of chronologies do not directly indicate the order in which lines 
of code were enterfed by student} the code was recorded .on the chronolc^y ' 
When the student listed or ran his progr'as and it was possible to 
deteniine saall changes in the code by coi^aziag successive listings. 
In the third group of chronologies, each tlee the student' typed a line 
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of code' it was written to the chronology and, in addition, when he 
listed and ran the prograa, the order in which the lines had been 
entered was stored with the listing. Since we' found this Inforisation to 
be useful, our analyses focused prioarily on the third group of 
chronologies* 

Figure 8 is an excerpt fxon-& chroDolog}^^^ In gener^, 
^chronologies record che sequence of BIP consands and lines of progran 
code typed by students vhen they worked on their programs. The cotmands 
include: 

LIST - lists the student's prograa 



RUN 

Dim 



executes the progras 

executes a oodel solution stored for the task the 
student is vorkiog 
^ prints a hint stored for the task 

lEACE - executes the student's progrars and prints for each 

line the values of any variables that changed, 
FLOW - executes the program one line at a ti^^, shoving 
' " hov variables change, and using the CRT to indteste 

TP ' the flow of control graphically • ^ 

- executes the progr^ and the Do4'el solution on ' 
test values and. compares their output iq order to 
judge whether the student's progran is correct 
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Lines of code entered are denoted by the keyword LIKE, or SYhTAX ERROR 
if the student typ^ an incorrect line that could b4 detected by the 
parser. E^ch entry in a' chronplogy includes the tiise at which the 
coEsand was typed- by the student. It does not alvays include the exact 
response of BIP to that consand. For exa^le, while LIST does put the 
progran listing on the chronology, HINT only puts the hint nueber, not 
the texb of the hint- 

Ibe chronolo^ data constitute an indirect window ontQ students' 
reasoning as they designed and debugged their {rrograos. For exat^le, if 
asCudent ran a DEKo'after partially coding his prograa, it sight be 
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run 



5/9/77 10. 16: 51 . / 

same program ^ ^ 



same program ; ^ 

output: TYPE IN TWO OSD NUMBERS* THE LCWER ONE FIRST, 
input: -1 ^ * 

input: 5 

output THEE& ARE 7 NUMBERS' BETWEEN THEM. 

completed at line 42 ^ ' \ 

*■ 

f 1 (IU» , 
5/9/77 10: 17 17 

output: TYPE IN TWO 'ODD NUMBERS, THE LOWER ONE FIRST, 
input;, 21 ' ^ 

input- 23 

output: THERE ARE 25 NUMBERS BETWEEN THEM, 
aborted at line 42, 



list 



1 ine 



detno 




c -' ' 

5/9/77 10: 19;D9 ■ 
same program 

I' 

5/9/77 10: 19; 42 - " ' ' * 

41 PRI.NT "THERE ARE/.C, " NUMBERS BETWEEN THEM" 

5/9/77 10: 19: 47 - . , . ' . 

orrf^r program listing ^ 
30 01 C=0 - ' ■ - 

19 10 pRIhfT -TYPE IN TWO ODD-IMUnSERS, THE LOWER ONE FIRST " 

2 15 INiPUT L. H * ^1 

3 20 IF L=« OR L>H THENF' 100 

7 26 IF L/2-0. 50INT(L/2) THEN 1 50 ' ' 

20 27 IF H/2-0 50INT(H/2) TWEN 155 ' 

9 30 FOR I=L TO H STEP 2 ■ 

10 35 C= C+1 

11 40 NEXT I ■ 

3f 41 PRINT "THERE ARE C; " NUMBERS BETWEEN THEM" 

27 42 STOP 

29 100 IF L>H THEN 110 

13 - 101 PRINT "YOU TYPED IN THE SAME NUMBER TWICE, TRY AGAIN 

WITH 

14 102 PRINT ."DIFFERENT N-UMBERS. " 

18 103 eOTO 15 ' ' 

16 110 PRINT "YOU SHOULD TYPE THE LOWER NUMBER FI^SJ, TRY ^ 

/ * • AGAIN. 

17 ill GOTO 15 j . ' 

25 150 PRINT "THE LOWErI NUMBER WAS NOT ODD, TRY -AGAIN " 

23 151. GOfO 15 \ 

24 155 PRINT "THE HIGHER NUMBER WAS NOT ODD. TRY AGAIN " 

26 156 GOTO 15 ■ . . 

28 999 END 

ft '. ' - ' 

input 21 

input: 23 ■ " . .. 

Figure 8. Exceipt free a BTP cinj^ology. . The student vas debugging 
a prcgraa at this point. -w*--* 

» • ■ ' : 
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(Figure 8 Continued) 
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5/9/77 'lO; 21: 03 ' - • ' 

, 41 PRINT "THERE ARE "iC." NUMBERS BETWEEN -;L AND ";H 



5/9/77 10:21:08 ... - 

order program listing 

1 01 C=0 

2 . 10 PRINT -TYPE IN TWO ODD NUMBERS, THE LOWER ONE FIRST. " 

4 20 IE L=H OR L>H THEN 100 ' 

5 ' 26 IF L/2^. 50INKL/2) THEN 150 

6 27 IF H/2-0. 50!NT(H/2) THEN 155 * - i 

7 30 FOR I=L TO H STEP 2 
S 35 C= C+1 

w 40. ^4EXT- I , . • . - ^ 

ori^f^ "^^^^ "'C' " NUMBERS BETWEEN L ; " AND »; H ' 

12 lOO IF L>H THEN 110 

13 101 PRINT "YOU TYPED IN THE SAME NUMBER TWICE. TRY'AGAIN 

14 102 p'rINT "DIFFERENT NUMBERS " • ' 
' 15 - 103 GOTO 15 

16 110 PRINT "YOU SHOULD TYPE .THE LOWER NUMBER FIRST, <% 

17 111 GOTO 15 ' " AGAIN. " 

15 150 PRINT "THE- LOWER NUMBER 4^/<S NOT ODD, TRY AGAIN " 
151 GOTO 15 

21 156 GOTO^ HIGHER NUMBER WAS NOT ODD, TRY AGAIN. ^ 

22 999 END 

output- TYPE IN TWO ODD NUMBERS, THE LOWER ONE FIRST ' 
. input. 21 i' 



output THERE ARE 2 NUMBERS BETWEEN 21 AND 23 
completed at^line 42 



5/9/77 10 21 32 
same program 



f//9/77 iO 21-45 
01 '€.--2 
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that he has become confused about the problea or a^ternatiyeiy that he 
is designing and iapieaenting the prograa in sections and is ready to 
exaaine'the requireisents of a new section. Looking past the DffiO in the 
chronology, if the student changes, code he had already entered, we would 
opt fof the first interpretations ft he ^nterdd soae new code, we would" 
choose the second. Tracing through a chronology and trying to 
re^nstruct what the student's strategy was reseables the task of an 
/rchaeologist working to infer the values and tsotivations of a society 
fron physical artifacts".^ z 

Our initial exaaioation of the chronologies was directed at 
identifying debugging episodes- involving logical bogs. These are btfgs' , 
that J.et the program execute but cause it to produce Incorrect output 
(e.g., GOTO an incorrect line nunb^r), as opposed to those that are 
syatactic or context-free and- are detected by BIP's parser or runtiae 
systen (e.g., GOTO a non-existent line ninsber). Further, we looked for 
episodes where it seemed that the student considered the prograa to be 
completed at the time- he detected the bug. as opposed to episodes where • 
the debugging seeeed to be integrated into design— i.e.. where the 
student was trying to di'scover^how some unfaailiar prograaaing construct 
worked. These distinctions had to be inferred by looking at bow the ' 
program was ceded and by how readily the student seemed to change the 
prograa. One sure clue that a student thought a prograa was complete 
was his calling BIP's solution checker with the MORE coiasand. Since BIP 
requires that a student RUH his.,prograQ before MORE will be executed, 
typing MORE iaplies the student had RUH his prograa and thought it was 
correct. • . 



' ^ has the saae potential pitfall that the researcher's own 
world view restricts the interpretations he night see. 

^ ' 57 - 
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Most of BIP's programming tasks Inyolve Interactive programs 
that process input from a "user^" Jin programs where flow of control was 
conditional on User input, we found many ^isodes where the student ran 
his program with inputs that did not caxise a bug tos^manifest itself and 
then typed M3R£. In most 'cases, the Inputs used by the solution checker 
did detect the bug, although sometimes the solution checker incorrectly 
accepted a student {urogram with a bug in it. Thus,' it became evident 
early in our ejcamination of the Chronologies that many students did not 
recognize the need to test programs across conditions that would 
exercise the different branches of conditional control structures. 

Our ori-gfhal plan was^to analyze the debugging episodes we found 

.by parsing them with context-free debugging grammar sialAar to that 

found in Killer and Goldstein's (1976b) (Figure 3) planning/debugging ' 
r 

grammar for LOCK) prdgra^ng. One feature of the contexts-free grammar 
rul^s is that a particular higher-order node (left-hand side of rule) 
may be expanded i^terms of alternative lower-order nodes (right-hand 
side). One of our goals after parsing the episodes was to examine 
alternative expansions of a higher-order node to determine what 
s^aantics of the context determined the choice among alternatives. Thus 
if there were a rulej 

repair :» replace-code | modify-code 
(the "M" is read as "or") we would be looking for features of a context 
that predicted when-e bug was repaired by replacing old code and when it 
was repaired by soma minimal editing of existing code* By determining 
the semantics,* a ©ore\owerful ATN graasar tfien could be developed for 
describing the episodes. Once the general grammar describing the 
debugging strategies was formlated, our plan was to characterise the 



differences between students in tenas of alternative subsets of the 
general graasaar. they eisployed and. In particular, to see If the poorer 
debuggers Vere those with degenerate versions of the debugging gramnar. 

The effort to derive a ^raaiaar encountered probleas lEHaedlately. 
At the lowest level,' where we were trying to Identify rules mapping onto 
the chronology keywords {RUN, LIST, DQIO, LIHE, etc) and the tlaing 
_ infonaation, we found an unexpected degree of variability bath within 
and between students. For instance! by exaalning several episodes we 



might derive 



test-repair RlftM;<loug latency> | 

RUN 'f^fenftlatency> + test-repair 



(the is r^d as ''acrd^then") a general rule of the gransaar. . 
However, in soae other episodes we would then observe students changing 
a line of code, listing the program, and fhen. changing that line again 
without ever having RUH the progras to test the first repair. We soon 
realized that because the programs were on the average short (a ©axisuta 
of about 30 lines) that student' sight have been testing the programs by 
loolting at a listing and nentally tracing its execution rather than 
running it. We could have added this alternative to the rule for *^ 
test-repair, except that USX t <long-latency> occurred in other rules 
as well. In fact, LIST following a repair was a cossion "cliche" in 
students'- behavior i evidently each tise they changed soae cade, aany 
students listed the program and looked at it briefly, sisply to verify 
that B^F had inserted the code as they intended* Although, the tlse* 
-spent for such a visual check is less on the average than that spent 
mentally executing a progras, the observed tiaes overlapped enou^^ to 
oake use^f the time data to distinguish these cases- imreHable^ In 
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other contexts, LIST often did not occur when it was expected; we - 
hypothesize that for shorter programs/ an earlier listing could still 
have been on the CRT after a few intervening events. .Thus,J.IST was not 
a reliable indicator of when tfte student had been examining the progr^y 
and when it did occur even examination of the surrounding context was 
'insufficient to determine the type of thinking the student was engaged' 
in. It soon became clear that even the lowest level rules in the 
debugging grammar would be complicated by alternative and optional 
patterns of keywords, and that the same patterns would be included in 

several rules. In most episodes, the only way to piece together what a 

I " 

student's strategy had been was to integrate semantic clues, from 
tjiroughout the episdcf^^ and even that involved making sometimes tenuous 
inferences y We found therefore that it does not seam possible to derive 
reasonably debugging grammars, of the type proposed by Miller and 
Goldstein^ for describing a range of episodes" in the BIP student 

chfong^lo^ies. 

/ 

^ven though we were unsuccessful at describing the debugging 
strategies of different BIP students in terms of a unifying 
information-processing mod^l, the episodes we exaained were Very 
infom^ative wi^ respect to identifying weaknesses in the debugging. of 
thesfe' inexperienced pro^amers. As' we mentioned , there were frequent 
failures to test progratas thoroughly ^en they were first f^un^ This 
failure generalized to testing after repairs as well. Although there 
were ambitus cases, in most instances the subsequent context made it 
clear that RUNs we judged we should have found, were not being replaced 
by nental execution of the prograa. As a result of Inadequate testing, 
students failed to detect bugs in their progtaas.' Hot surprisingly, we' 
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4180 observed that even wh.en students did detect bugs by runhing their 
prograi^, they did^^not rerun the' program with varying 'inputs, whlc^ by 
, exercising, different parts of t-he program's control structure would 
produce output us efi4;f or localizing, the fiar'tSf the prog;:am containing 
the bug (i.e., they did CHABACTERIZE)-..- ■ ' 

One of the most striking failures to test and c>»aracteri;?e thfit^ 
we foundfen the chronologies involved thfe program shown in Figure 9. '1^. 
is one student's attempted solutloQ to BIP's task CALCULATOR,' .which, 
* specifies 4|^ateractive prpgram foSr (I) getting two numbera from the ' " 
. user, (2) gett^ing a numerical code cc4?t?espond4ng to .one of "th^ four 

V . J • . , . 

-•primary arith«6tic operations (+, /), and (3) priating t:o the 

terminal the result of applying specified operation to the two 
numbers. The _student 's program has a fundamental f low-of.-control 
bug(s), which results 1(1 execution, "falling "through" the code fdr 
computing and printing the results (lines, 80 to ,150). Thus", for 
example,' if" the user typed 'a "1" to 'specify add»Ltion,. the program 
branches correctly to line 120 to c^" the, ^ddltlon, "but incorrectly 

■•continues on to do subtract^, multi-pli'ca^ion, and division. 
Similarly, for subtraction, ^^he multlnT^a'tlon and divisioo are computed 
as well-.r Qgly foe dlvifflon, the^^al branch in the control structure, 
does the cglciilator compute apd-^p^lnt^onl-y what it is supposed to. Tq 
correct ttj<l" programT'tTrre^i^ii^, 'l25,. t35, Snd 145, all of -which should 
be STOj/or GOTO 199,. must be Inserted. .The Student who wrote the 
pro-am failed to. debug It; in fact, he ^lled^to de.t^ct the' bug at all, 
although he-called the solution'checker and had it reject the program 
six separate tines! ' -.' ' ■ . 

■the major cause of the stadent*8 difficulty was tiiat everjr tlffle. 

i " # ■ . * 
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10 PRI^JT ''THIS IS. A CALCULATOR." 

20 PRINT "TYPE i TO ADD, 2 TO SUBTRACT, 3 TO MULTIPLY, AND 4 TQ DIVIDE" 
30 INPUT C 

40 PRINT "NOW CHOOSE A NUMBER"' . ^ . 

50 INPUT X . ' / - • ■ 

60 PHINT "NOW*CH00SE ANOTHEff^UMBER" 

70 INPUT Y 

80 IF C = 1 THEN 120 ' , 
90 IF C =•• 2 THEN 130 
100 IF^ = 3 THEN 140 ' 

110 IF C = 4 THEN 160 . . • 

12»-t?RINf "THE SUP1 IS "r X '+ Y 

130 PR,INT "THE DIFFERe'nC^ IS *VX - Y i 
140 PRINJ "THE I^RODUCT IS " , X** Y <" " ' 

150 PRINT "THE QUOTIENT IS "i X / Y' • * . 

199 END . • . 



Figure- 9^ , A ^tudent^s, solution to BIP*s. task CAICDLATOR. Tte program 
has a recurring flov-of-ccaitarol bug, which the student 
failed to detect because of inadequate 'testing, * * 
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be typed/Wm to 'test his prograi»r'"R&N8pecif ied /*4'\ as the operatioji 
code. Not once did he test it with another operator* Since, "4" for 
division is the only case in wh'ich the program works cor.rectly/ he never 
saw the bug manifest itself in the output* In b^ween running the 
program and calling the solution checker, we found that he used LIST and 
spent 'long periods before his next RUN. Assuming that he looked at the 
listing during these periods and because three similar lines were 
missing, we can conclude that he did not understand hdw to design 
conditianai control structures and hfttTnot dimply made a careless errors 
However, if he had RUN the program just opce with ^ code other than 
the erroneous output could have served to help him understand the defect 
in his design. • » . ^ ^ , . 

We found many other examples of inadequate testing and 
characterization- In lact, there was evidence that evea^when they raiT 
the program and it produced incorrect output, soiae students did not 

realize that there was a bug in the program. In these cases, the 

'* * * 

students called the solution checker immediately following their RUN, of 

4 

the program, suggesting either that they had not analyzed the output or 
that they did not understand what the prograa they wrote was supposed to 
^do»^ Based on independent observations we made, we believe that in oatry 
cases the students did not anaiya:e the output. A sender of the research 
t^m spent about; 20 hours observing (and assisting) cowse consultaivts 
and student? discussing problems for the introductory ALGOL programming 
class "at Stanford* These students probably have a higher Aptitude for 
programming on the average than the BIP students and work on prograsaiag 



The solution cbeck^Tr at that time did hot attempt to tell the 
student vrtiat it had foun4 wrong, so that it was nOt .called as way to 
obtain information. - . * * 




tasks more coaplex than those in' the BIP curriculua, . Thfey usuajX^came 

r 

to the ^onsuitants vhen they had trouble, debugging their 
^ ' large number of cases, students, had not looked at their t_.^_^ 

'than to note that tbe^rograa did not work. Instead^ t;hey des^^e^ 
their debugging as going through the program line-by-line looking for a 
mistake, even though they had not thought about what was wrongs For " 
errors trapped by the ALGOL runtime systeia (e.g., illegal Detaory 
reference) their debiigging was even tsore irrational, since they did not 
attend to systea diagnostics which could have identified the type of 
statenient cpntaining their error or, in sotse cases, the actual line 
containing it* Thus, the general debuggttxg strat^y of ' the ALGOL 
students we observed was -deficient in testing and characterization in f 
much the saae way as that inferred fron the chronologies of the BIP 
students* 

^ Another type of poor debugging strategy we observed in the 

chronologies involved students aaacing a series of several minor, 
sooetises completely 'non-functional, ©odif icatiops to their prograss in 
^. a very short period of tise, Most'often, this behavior was seen in the 
same -episodes where there was no attenpt to characterize the bug by 
running the progras vith vailing inputs, A related failure was that 
' attecapted repairs that did -not correct a bug were not undone at once and 
. * evidently wei^e forgotten. As a*result, "almost .correct" progrtes 

sota|tiQ^s were rendered less correct during debugging as the student 
coi^qunded the original^ug with others resulting fros the ineffective 
repairs* 

In order to subs tan tiarte -some of the inferences we had drawn 
*froia the chroSj||logieS;^ we collected** written deltoggiq^ coaaentaries fros 

^ - \ 64 74 

ERLC 



inexperienced prograaaers working on staged debugging probletas. The 
procedure was siaiiar to that under which th^ coaaentaries were obtained 
f rem _programBer8 debugging the sorting prograa. Four students who had 
conpleted 10 hours in the BIP course about one-half year earlier 
participated in the study. Each worked to debug a series of programs 
within the BIP prograaaiag eovironaent. the prograas theaselves were 
selected froa the chronologies arfd involved different types of bugs: 
conputacton, assignsent, flow-of -control. This aeant that the students 
were debugging progp^ written by other inexperienced prograaaers as 
solutions CO problecs they had theaselves atteapt.ed in BIP. 

The students were instructed to aaintain a written record of 
their thoughts as they tried to debug the prograas. In particular, they 
were, told that whenevef they decided to take an action— LIST or RUN the 
prograa, or sake a repair— they should record their reasoning, BIP ■ 
cbronofogies were saved for the debugging sessions and in addition the 
sessiops were conducted on ijardcopy terainals, instead of CRTs, so that, 
exact typescripts' of the interactions could be obtained. For each 
debugging problea, the stuatefits were given a listii^ ot the prograa and 
•a description 'of what is was supposed to do, but were glv^n no saaple 
input-output data. A copy of the firograa was. preloaded into their '* " 
prograa space in BIP, so tba^jthey theaselves did not lave to type it in 
in order to ruh or aanipulate it. Ihey had at sost an hour to work on 
each problea. " ■ ^ 

Ip d?«ettbiijg the results of ^Ms study^two geieral 
observations oust first be 'noted. |he subjects bad not done «ny ; '* ' 
prograaaing since the tiae they finished BIP and their behavior, and the ' 
coaaentaries indicate they bad forgotten features of the BASIC language 
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an4 of how %o use BIP. Therefore, such of their effort, especially on 
theg first few problems, was spent using the BIP aanual and trying to 
relearn fundaiDentals. Second, the subjects had trouble eaintaining an 
ongoing cotssentary, Tney would work on the problea for a while and 
afterward write, rather than write as they were thinking, Tne 'observer 
provided constant pronpts^ to r-esind then to write and they were 
encouraged to write and not to concern thenselves with working quickly, 
^Nonetheless, the ecczDentaries are fragyseatary records at best and are 
Dore retrospective accounts of what the subjects were thinking than they 
are real-tise records. 

The cocrentaries substantiate and elaborate our observations on 
the inadequate debugging strategies we saw in the earlier chronologies. 
Again, the i^st salient deficits were in testing and characterization, 
in obtaining and using inforriation fron a bugged progras^s input^-eut-put 
relationships • Fron the c(x^&entaries we could determine that when 
students listed and exaained a prograri, they were not substituting 
nental execution for' an actual cot^uter run, *but were scanning 
individual lines of code for eri;ors. H0W7? since the subjects were 

- , _ ft 

debugging programs written by someone else, it is not necessarily a bad 

strategy to list and exasiae the b*igged ^rpgraa in a' global Vay in order 

* to detersine its overall organization- However, there were several 

cases in which subjects repotted losking at lines of code for errors, 

when they had not yet run the program and seen how tfafe error eanifested 

itself, as illustrited by the following excerpt: 

I've done this progras before^tso \ feel confident 
that I'll be able to find the bug. After one reading 
I've no idea lAat the probles is. I just J.ooked at 
the two input stateseats, they look 0K# t just looked 
at the 50 statement •* Ho thing looks wrong there* - 
run the program to see if there's a problem* , 
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1 just- read the output on siles pet gallon. Thought: 
I^ve got it! The machine divides before it subtracts. 
I'll try -putting in parentheses aroiaid E-B to see if 
it will subtract first. 

This subject cecogoized the. progras ami thought he coiiid find the bug 

just by looking at it. He examines the. prograa listing llne--by-iiae 

without- success. Then he nms the progras.^ sees the nature of the 

error, and 'is ii^zaediately able to locate the bug. 

I^e cosr^ntaries Indicate the sechaaisBi for the quick and 

apparently unootivated repairs ve had seen students sake in the 

chronologies. Consider the follovln| excerpt: 

rnis equals business in 160 td 230 is ccmfusing 
stuff. Seecs to se th^'re Rouble assigning 
things. H and L are being gives two values* 
I think saybe 160 and 170 can be deleted. 
Try and see. 

ihe subject, without having run the prograa, examined the code and saw 
socething that looked "confusing." Consequently, without any sotmd 
reason for doing so, she deletes two lines. This co^ounded the bug Itf' 
the probleo, so, that when she tested the prograa (for the first tiae) 
after the repair and it worked incorrectly, she had to go back and uiyio 
the repair and run the progras again In order to see the s^ifestatlon"^ 

-of the original bug. 

One of the subjects did sees to have an eff^tive t^-^own 
strategy with el^nts sisilar to that of our troubleshooting/debugging 
©odel. However, even he had dlff^^culty because his CHARACTERIZE 
procedure was not well developed. Figure 10a is a ccK^lete ccwentary » 
froa this subject for the bugg^ progrmi shown li| figure lOb. Be reads 
the prograa first, but only to Identify its structure. He then rtms the 

' prograa, but happens, to* choose Inputs for which the prograa works 



V 



SIP 



Tols prc^ra= locks scajy because it's so Icaig. I'e goiEg to try to analvze 
TJ^J^. fr^"^' denoted e^tr^T^t Z-^T^ 

tiff 30 I doa't ui^rstaM. I'll loc& it up vfaen l4 dcos r^adin^. ^Tl 

to 4^?rs=.r^iLa?- ^^^^ - 

■ • * i 

TaJZ "^"^ that is the l±De between voiking aad mrt ■ 

vozking. (3) I'll tiy distances of 2, ij, etc. 

th^'priblS.' ™^ ^""^^ ^ sacMiie paid that's prol^ST^ 

f ^ infinite Icx:^. I'ul ioc^ ^tbe naabers for a -.^ 

aad see if I can figure anything oit of that. Well, P stayed tie sa=fe n St 
S'S^ ^ ™ f"^^ see vhat ^aTl St^cid'SL 
SaSngn^a:^^sr-^JS--S=^l'^ ^'^ 

up a:^Lf "^'r^s^f '° ^'"^ ^ ^ ^ ^ 1^ : 

Gc^aldn't solve by I315. , 

Figure 103. Debuggicg ccraentaiy of aa ilisxpeTiencaa pziDgi^saer ' 
atte=5»ting to debug the progrss BlKnra in Figure lOb, 
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ii-^ user inputs two unequal odd nuobtrs (tba program checks -to make sure 
-f^at this IS th% case and a^ks the, user -to try again if a ©istake has 

^een made). Odd numbers betu»en his tao numbers, inclusive, are counted 
-P^or^exaople. there are 3 odd numbers betsaeen 5 and 9 — tbfey are 

y. and 9 Finally tfte number' of odd numbers iretween the oser's two 

^■jnoers is prirjfced. *' , ' / 



Oi/01/77 00: 00: 01 - 
2-7 . " 

10 PRINT "TYPE AN ODD hfUttBER" 

20 irOPUT ^ . ' ■ ^ 

30 IF. X/2 O INT(X/2) TWrn &0 ' . ' . 

-40 PRINT "THAT IS NOT AN ODD NUflBEH. TRY AOAIN* - 

50 GOTO 20 . ■ ■ 

60 PRINT "TYPE ANOTHER ODD NUJ1BER" < 

70 INPUT Y , '■ 

SO IF Y/2 O lNT<Y/2) THEN 110 ' ' . " 

90 PRINT "THAT IS rtOT ODD NUMBER. TRY AOAIN " 
lOO GOTO 70 

110 IF X O^Y THEN 150 • • ^ 

120 PRINT "YCUR TWO s'uflBERS ARE EQUAL. TRY AGAIN, THIS TIME" 
:30 FRINT "USING TWO ODD NUMBERS WHICH ARE tSOT 'EQUAI_ - 
140 <tOTO 20 ■ ■• 

150 IF X < y THEJ4 1?0 - ' * , 

H = X ^ 
170 L = ?. . 

130 GOTO 210 ■ . 

190 H = Y ' ^ 

200 L ^ X * . , • 

2t0 N = I • ' 

220 P -= e + 2 . * 

230 N = N + 1 
240 IF P = H TH£N 260 - 

250 GOTO 220 ' ' » • 

799 ENO*^' "THERE. ARE N; - ODD hVmERS BETWEEN " i U " AND -j H 

\, 

4s, 

. Figure ICS), Bagged soltitits to SCP's task OIXXHJHT, used to study 
debagging by iiaszperienced prpgraasTB'. fbe fejg is in 
Liije-^ vhich sboaJd-be P =•? + 2. In addition, Mne 
. . 210 Eust be P = L aei" Idm 215 caist b^ E = 1. 
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correctly and-l)ecoaes '*scuck/* Only a prompt frois tli^ observer induces 
hiQ to try other inputs and thereby detect -the bug. He arrives at a- 
correct ACTIOH^ESCRIPTION that the prograo works correctly only if the 
pai,r o^ numbers are consecutive. He does ndt debug the prograi^ within 
the jtine allowed, but this can be attributed his forgetting sone of the 
BASIC language constructs needed to underst^and the function of parts of 
toe prog^an. 

,The effective strategy of the saise subject can be seen in the 

following excerpt in which he wa6 debujsiginR the prograo shown in 

Figure 9. Hote nis careful initial cn^acterization and testing 

following repairs, and h5w he resists juisping to conclusions until he 

has exanined tne program's output* • 

I just read SID (the progran). I just thought the 
problen ziay be there's a problea with end or stop 
statements. I'll run the progran to have a look at 
it. hfy susp icio n see^zed correct. Tne cal^ulato^' 
outputs alL f uncrTloos* sc^^-I 've 'got to find a way to 
licit the calculator to Its assigned function/ I'll 
look in the glossary to ^ind the right word. I 
couldn't find anything so I'll cry GO TO statenents 
after eacH function. They'll say: Gu TO,., end. I 
JuBt typed a 125 GO TO 199 statexSent. I'll now run 
the addition ^d see if it stops. 2t worked. I was 
pretty confident it would. How; "I'll add these expressions 
to t^ other functions* I nade a mistake in typing, 
so I'll look up th*e CTL button for_offing a line. 
found it, I'll CTL X. How,r lUl i:un again, checking 
all the functions,^ It worked. I want to try TRACE 
now. Just to make sure I understand it.. 



Our observations of debugging by inexperienced prbgraaf^rs ' 
support the hypothesis that sotse of then have difficulties not only 

because tlmy are not well-versed in 'programing fundamentals and lack 

> • * ^ 

libraries of specific experiential knowledge, but because they have 
Inadequate general debugging strategies. In Articular, the are 
defici^it ip^ running a progras to obtain inforaatlon. that can be used to 
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deduce logically where a bug is located. In addi.tion, they make repai 
without good reasons and lose track of repairs they have atteapfced, 
thereby confounding their problem.- 




III. Teaching Troubleshooting/Debugging 

> 

Isiproving Instruction in coctplex probleQ-solving 

In the introduction, we described the indirect isethod by which |^ 
croubleshooting£debug£lng and other typ^s of complex probleo-solving are 
currently i;aught. We mentioned two problems with this laethod. First, 
in domains where prabiea solving requires specialized facilities, such 
as electronic troubleshooting, costs llnilt the range and nunber of, 
exanples and exercises- students may experience during fcnaal 
instruction. Thus, students of average or above average aptitude may 
not have experience' sufficient for then to acquire probiecj-solving 
conpetence. Second, students with lover aptitudes may have fundamental 
difficulties learning by. t-he indirect method even when a relatively • 
broad range of experiences**can be provided • ^^^^ 

One solution to the first probleo, and perhaps the second, is to 
elaborate on rhe indirect approach in ways that can increase student 
exposure to problem-solving experiences and add structure to these 
experiences by providing core and better feedback to ^in, A landmark * 
example of this' type of solution is the SOPHIE system developed over a 
period of several years by Browp, Burton, and their colleagues^ (Brown & 
Burton, 1975;- Brown, Rubenstein, and Burton, 1976), whiih provides 
instruction in electronic troubleshooting. Through the use of computer 
simulation and other AI, techniques, SOPHIE creates an enriched 
environment in which students may acquire both a general troubleshooting 
strategy and domain-specific knowledge for understanding interactions 
between pajts of -circuits, SOPHIE does have its limitations— mosV 
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notably, that all i^s exercises and ponltorirtu capabiliCl^ are liaited 
to a Single- circuit— but these are overshadowed by the advances it 
, represents in teaching by the indirect method. 

SOPHIE. The basic SOPHIE systea is ^n interactive 
coaputer-based t^ubleshooting laboratory built aroJnd a simulation' of a 
non-trivial power supply circuit. All s'tudlnt activities require only 
the siaulated circuit and no teal circuits oc teat equipaent. In 
var^oas operating oodes, compo<ients in the similated circuit can be 
faulted as "specified by a hunan instructor, by the student, or randomly 
by SOPHIE itself. The student Bakes meisureaeDts on the faultfed circuit 
sinply by requesting thee; they are detenainpd by the siaulation. 
SiDilarly, he specifies repairs by requesting SOraiE^o replace a 
component. These interactions are facilitated by SOPHIE's liaited. but 
very habitable, natural language frcmt-end, which relieves the student 
of learning a special language for cotaaunicating with the systea. 

■ " -In a basic operational node. SOPHIE allows Individual' student ' 
troubleshoor'an unknown Jayic or investigate the effects of a fault 
he himseif has specified, cmch as he alg-ht in a noraal circuit 
laboratory. However, it eliainates aany of the peripheral probleas 
involved in setting up and using real circuits and test equipaent. 
Beyond this^ SOPHIE constantly performs two powerful aoaitoring 
functions as the student works with the faulted circuit. First, before 
perforaing a aeasureaent requested by the student, it deteraines wbethec 
the requested value is redundant— i.e., whether it can be deduced 
Hogically froa the aeasureaents that have already , been "Bade— and. If \ 
.so, refuses to make the. aeasureaent. = In this way, SOPHIE alerts the 
student that he has sose aisunderstanding of the structure 'and teleology 
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oftphe circuit* Second, when the student asks that a part be replaced, n 
SOPHIE 'determines whether that part being faulted is consistent with the * 
taeasureaents that have been made. This is accon^lished by faulting that 
component in a copy of the simulated circuit, making the aeasarementrs ^ 
the student had made, and comparing them with tfie values obtained from ^ 

r • 

the version of the circuit the student is working with. "^If the 
specified repair is inconsistent, the student is told so. Again, this 
alerts the student t;o problems in his reasoning and understanding of the. 
circuit* 

In a second, more recently developed operating made, SOPHLE 
provides the student with "real-time" examples of" expert troubleshooting 
involving the power supply, circuit. SOPklE's "articulate expert" is 
programmed as blocks of hunahrauthored text that are accessed 
dynamically in examples involving different faults* Each episode'begins 
by the student selecting a fault he^would like the Expert to find. The 
faults are specified with respect to their location within the circuit's 
top-level functional organization* A "demon", another entity within 
SOPHIE, inserts the fault in the circuit. The ExBfiJt proceeds with a 
commentary on isolating the f auU vhich exes^lif ieb'an optimal top-down 
strategy similar to that in the troubleshooting/debugging laodel we 
described in the previous section of this report {Figure 11). 

Rather than directly, sake measurements on the circuit at each 
step ifi the localizat^^on pr'ocess, the Expert asks the student to tell* 
him the qualitative value of the measure he wants to^oake (e.g.. Is the 
voltage at point A high, I'bw, or normal?). *Thus, tfie student must 
understand the 'effects of the fault he chooses (a device for motivating 
the student to understand the circuit) ♦ The students answer is checked 
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How does the OUTPUT VOLTAGE compare with a working 
circuit? . . / . , 



DICE: LOW ' . 

EXPERT: ' ^ ' r . , ^ • -r 

pvery functional blo^k has a failyre ^(Jde - which Bxp^ Itjwa of 
could ckuse the ^output to be low Hinder these fmlts in evsry block 
condition^ so I need some more information! 

Isf the OtftPUT VIQLTAGE' essentially zero? 



irfiicft cause 'Im output, 
the only synptom eo far* 



DICK: YES 



57ze Expert asks fbr a rrore 
refiMd, althoitgn^etill 
-qualitative prediction 
dbout the output. 



EXPERT: . . " ' 

AH GOOD! Thai is impprtant information *f or me 
because I don't suspect the Vr^ef nowl ' If the Vref 
were -dead, that is/ had no output whatsoever, then ^ inatayy^e of teleoXo^ 
the output voltage would still be around one antT'lrf^?^^ hwuUd^e^ at a 
one h^lf volts./ It would have this magnitude""^^ leve^l: failures 
because the voltag^ liiaiter cannot evpn begin to in Vref camot trgke the 
function until its input vol^a^e risas ^^afe«v% * the output j^sentiaZly zero. 
.junction . voltage drop 'of its^lnputr^rarisistor This is a sormjhat ' 
And, of course, if Vref ha4" any voltage output Surprising fact vhicf^ the 
gra#ter<^ than„ zero, t)ie output volt^^ge would be ^t^lhit my /knt to verify 
Still greater>f. Having, el imlhiitJed the Vref,,!* now ^^runentalij^ in the 
wonder about, the CL, Vt'rDfCtd, ^ and OPF-* Any SOPBlE lab. 
one c^f thesife, could cause- the output vol tage^^to bfe 
zero so I ^ going to -narrow the se^pph. \, 

How doe^ the VOLTAGE AT.Nl co^pariTwithV^ working tttZtWft this test is 



aimed primccrily^at only 
* ^t^i^^^kj it iq nonethe- 
" Xes^jkeirhble because 
^ 'the rest of the blocks ' 
cannot ^imction unless 
this blocks is ' working r 



FJgu'ife 31-. 5xQerpt of an ^interaction wlth'SOTHTEJ, 

,..^Dc& Brvun, I^ibensteiii/|ind Burton^ 1976^ 



' ^^articulate* ^expert% " 



by the demon, who "invisibly" 'runs the simulated circuit to get Ih^ 
actiial quantitiiKive value. .If the student^s answer is wrong the demon 
interrupts an'd tel^ the .stu<^€h^^nd the Expert. The Expert follows a 

J strategy of'thoosing measurements that' enable him to reduce 
progressively the part of the circuit's £unctionaKor]jj^nization that 
remains to be considered . _After obtaining each cj^alitative measure from 
the st|f3ent, he explains how'l.t enabled him to deduce £hat the fault 
^ould not be in certain subcircuits. The 'Expert never describes this 
local(zati.on strategy in general terms; instep, the sfudent is left 'to 
induce^the general principles from the specif ic 'exatrifj;.es of ^reasoning . 

^rown, Rubensceln, and ^Button (1976) report a* study -txj which 
they evaluated the reactions of a small group of secon<f-y^r electronics 
-^students from a technical school to the SOPHIE system. Each subject 
interac^ted with'SOPHIE in several modes^ including the two we have , 

? described here. In questionnaires and interviews the students in 

>^ 

general indicted that SOPHIE _wa« superior to the-ir normal experiences 
. in a circuit laboratory. They believed that'thB individual^ ^ 
'troubTeshootljig. activity did teach t^hem knowledge that would be useful 



m troubleshooting other types of- circuits* Their criticism was that 
when they wereV^t^ld^about their* attempted' redu*iant measurements or - 
r illogical repairTs^ they .could not- always understand why the^ were wrong 

m 

and could^tain no fOrther help from. SOPHIE. . ^ 

^ "* . ' . . " 

^ - ^ interaction with the Expert was also rateid favorably, but' 

- » " ' ' ' s-^ ^ , . 

not as hrghly as the other conditions and^^^ith |g)re variability among' 

^tl^r 6onditton» included * competitive troubleshooting game 
between two-person teams, and an exercise in which the student had to 
speci&^ a fault Vhich w^en inserted in the circuit would ca^e another 
target^ cemponent'-to fail as well. . \ ' 
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• stiKients. StifdJjts who liked this condition -reported tl?at they believed' 

that by observftg-the Expert Chey had learned a general problem-solving 

strategy of top-dS^>n decomposition and testihg th^t they could apply 1^ 

a -range of problem-solving .contexts. . Tbe' students who r^ed the 

• s. ; • . ' 

: condition poorly tended to be those of loweV aptitude who had trouble in' 

\ - ' 

individual debugging and in answering the Ex^rt'^ questions. They said 
they found the expert too glib and were frustr^d by not being able to 
question hin. They had trouble juSt f oliowl^ h^s .comaenta^ on a 
specific problem, let alone being ablap>-lnjiW? the underlying, general ' 
strategy. (j / 

- The results can in general be taken to. indicate that the 

capabilities of the SOPHIE system can inpro.ve learning of both 

fimain-specific knowledge and general -itrategies within the indirexit ' * 

method of leaning by^exai^le and exercise. However^hese benefits^ 

accrue primarily fer -students of nomal or higher than -normal aptitude] 

and are dtiainished or lost on t.hose of lower aptitude- 

. The Why system . 'The Why system (Coll\n8, . i976; Stevens & 

Collins; - 1977) is another CAI system that a^y'be viewed as -a'n enriched ' 

context for q^hing proJJlem-solving by tfie indirect^ fiet hod . -The stated 
... , ' y ' " . '- . - ^ 

putpose of the system is to teach domain-specific dedlarative knowledge* 

u- . . • %■» • - 

about geography bj^ engaging the student .in A natural language dialog 

with a simulated Socratic tutoir (Figure, 127.- However, as we noted In 

the introduction,, tutoring can readily be'viewed as debugging of the 

student's knowledge system. - A tutor's behavior therefore' exeoplif^s 

the ai>plicat±on of a debugging strategy', .which the student piay induiQ 

.'and subsequently use to debug hims%lf. An interesting- Teature of Why is ^ 

that, unlike'^SOfHIE's articulate expert -.the Why tutor generates his '• * 



f 



. ■ - •■-"85 



DO YOU TiJiNK 'IHE AMAZON JUNGLE HAS HEAVY RAINFALL OR LITTLE RAI 
(Rul.e J: Ask for a prediction about a particular, case) 

*ritAVY Rainfall* ; 

WHY DOES THE AMAZON JUNGLE- HAVE HEAVy RAINFALL? (Rule 2: fisk for 
prior causes) ' „ 

♦BECAUSE tT'S NEAR THE ATLANTIC* ' 

. Y£S, THEi, AMAZON JUNGLE IS NEAR THE EQUATORIAL ATLANTIC. '•HOW DOES 
THAT AFFECT THE RAINFALL' IN THE AHA^ZON JUNGLE? (Rule 3 : . Ask for 
y i-ntermediate" causes) 

*THE WATER IN CURRENT IS WARM* " " 

(section omitted) ^ . ^ 

rthAT HAPPENS AFTER THE hOISTURE PILLED AIR ROCHES THE £AND? ' (Rule 
4: Ask for subsequent causes) , " ' " _ 

*i don't know* 

DO YOU IhInk the ANDES HOUNTAIH RANGE HAS ANY EFFECT ON THE RAINFALL 

IN THE^ AMAZON JUNGLE? (Rul^ 5: Suggest a factor and a"sk if it is ' a 
cause) • ^ " * • 

. *YES* • , ■• ■ . ■ - 



hQvt? (Rule ^3* Ask for intermediate causes) 

*THE PREVAILING wIND HITS THE ANDES AND THE AIR COOLS* " ' ' 

riOw DOES THE 'FACT tHAT {PREVAILING WIND 4flTS'^THE ANDES, .LEAD TO AIR 



COOLlNGt (Rule 3 ^ain) 



^section omitted) ' ' - 

DO YOU THINK^^ THAT ANY PLACE WIIH MOUNTAINS HaI* HEAVY RAINFALL? (Rlile 
6: form a general rule for an insufficient cause) • ' ■ 

*YLS*, ^' . 

SOUlhERN CALIFORNIA HAS MOUNTAINS. WHY ^DOESN'T ^ODTHBRN CALIFORNIA 
HAVE HEAVY RAINFALL? ^{Rule* 7: Pick a counter-example for an 
insufficient cause, and Rule 2: Ask for* prior causes) , ^ ■ 

; *Figure,12. Rotated excenrc of a "dialog with tbe Why system. Frcta 
Stevens and Collins, 1977. 
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questions dnd comnents dynaoically using an explicitly represented * 
process oSdel of_a general tutoring strategy (i.e.,^one that is 

'independent of the content of the sa&ject being ta'ught^. The sodel -is 
expressed as » production systeo of about two dozen rules {Figure 1.3), 

• which were derived by analyzing dialogs, between students and hua^ ' ' 
tutors. While this oodel underlies the tutor's behavior. It is no't 
articulated directly to the student and Is actually coacunlcated nore 
indirectly ^than the strategy underlying SOPHIE'S expert's "canned" 

.explanations. 

» 

More direct oethods for teaching stratefeies . A second approach " 
to -peaching coaplex problen-solviog, which sight help' those students who 
have Che nost diff flirty learning by the indirect lyjproach-, is to 
provide explicit descriptions of the procedures for solving. probleii 
that can serve as' prescriptions for the student. As noted In the* 
iatroductioa, an inpedijsent to this approach' previously . has'^een th^ 
lack of^a suitable .language for conceiving and talking abQut p.roblens 
and problea-solving processefB. The developsent in AI and 
infonSatiotj-processii^ psychology of forcalisas for representiiyj 
knowledge has caused researchers coac'erned with learning and. instruction 
toyteexa^e ttjjg; need g nd .pot^tlal for eore direct and explicit 
io^truccion i^^r^iez^-soiving/, \' ^ ^ I 

/ we uot attenipt- to teach 'soo^' basic * . * 

cognitive skills such as feow to organize one's " ' 
knowledge, hpw ,to learn, how to solve .j)roblems, how ' 
to correct errocs in understandings these strike us % 
as basic ccmponents which ought to be< taught Slong .* 
.with the content letter. 

Hoinaan, Centner, and Stevens, * I976,^^ p/ I$4. • , 

I 



RUL^2: Ask for an^ factors ' . 

If 1) a student asserts that a case has a -particular 
* value on the depende^*.t variable, 

, then 2) -ask the student why. 

.EXAMPLE: ^ ' ' ' \' 

If a student says they grow rice in Cbina'^^ ask why.* 
REASON FOa USE: ■ ' . , ^ / 

Tiris deterfilnes what causal factors or chains the' " 
. " Student knows about . . ^ . 
RULE 3: Ask for inf radiate factors ' ^ 

If 1) the student gives as an explanation a factor that 
is not an iiaraediate cause in the c/usal chain, 
then 2) ask for the int^hoediate steps. 
. EXAMPCB: ' * . ' , ^ 

If t>e^v^stj>*ent mentions nonsoohs in China, as a reason 
for rice growings, ask «Why do monsoons iaa,ke it possible 
to grow rice in China?*^ , 
REASON FOR USE: . ' ' ^ t 

This iHsures that the student understands the ^teps^ 
irf^.the causal chain, for example tliat rice needs 
to be flooded, * ' - X. 



R^LE 4: Ask 'for prior factors \ 

If 1) tHe student giyes as an Explanation a 

factor*, on. causal chain where there are 

.also%rior factoids, ^ 

then* 2) qsk th? student for the pMor factors. ' 

Figure 13 . Se-.^eral of the pixDauctloi rules Used in the Wly systen 
as a cocpitatioial model' of a tirtoring strategy* Frtn 
Stevens aM Collins y I9TT9 
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*.*as information-processing analys"^ succeed 
'in identifying «tbe processes underlying probles 
solution, these processes — at least some of thes— - 
can be <tlrectly taught, and that individuals will 
then be abl# to apply thea to solving relatively 
lajge classes of problecs. ways can be found 

to sake- individuals sore conscious ot the. role of 
environsental cues in pVoblen solving^ and to teach 
strategies of feature scanning an.d analysis. 

Resnick, 19/6, pp. 79-80. 

. V 

Papert (i97i) at HIT ^nas played a proaineat- role its articulacini? 

the position that by ceatehing general problee-solviq/ strategies ©ore 

directly, students can becc^e better learners. Bis argument is that 

learning co do things is facilitated by giving the learner a procedural 

representation of his task and having hi.a debug his' attecpt/^ execution 

of that procedure. Papert fefel that this methodolojo? applies to^sks 

as drverse as computational =atheaatj.cs and juggling. Much of iiis work 

nas mvQlved teaching cosputational satheoatics {prlaarily gedlsetry) to 

children by teaching thea to write 'prograns in the LOGO language. The 

stutJents learn tne natheaatics- by discovdiry {i.e.. Inductively) , but 

they are taught strategilb for desigb and debugging explicitly. The 

straffgies, however, ^re aot presented in coto . Instead, tHe cethod 

adopted is to present then- in parts ^s separate heuristics In reacflon 

io events that trans>ire "as the student designs an^ '(Jefcdgs hfs pregrans 

■ .-^ ■ ■ ■', ■ 

Tn .this context, a heuristic t&y be defined as a rule'-of-thutsbV 
^ I • » 

a piece of a larger procedure tbat eifables a correct or sore efficient 

solutfoij.under a set of conditions. The effec|:lveness. of -using a 

heuristic. ^epe^ds both *on being able to id^nb'fTy 'the fiontex^s where.it 

applies or is nore effective thatf pther ^eu^stici and |on access to 

other knowledgej^eeded to ^ecuce its' Heuristics aay embody either 

" ■ -'^ ■ ■ • ' • " 



general or docain-specif ic procedural knovledji€jt.^^^Jlie following are both 

heuristics for troubleshooting; hovefer* the first Is liaited to a very* 

specific context^, while tne second is part of the general strategy we"^ 

presented In the" previous secTion of the -report. 

i 

If the ^ar is idling unevenly, the first 

'thing to do is to strike the body of the ^ 
carburetor Vith several crisp ^but- 

non-daisaging) blows. * 



If you nave decided* to inake an observation 
of the system's behavior, choose th€ j 
observation' cnat has the potential to ^ <Cf 
eiininate the greatest part of the systen 
as a possible fault location whether the " 
observation proves to tfe nori^l or abnolrial. 



> 

In Fapert's rese^^cn studies, 5ne stu^ient's probl^a solvinj^ is 



continually ti^ditored by an instructor. When the stydent has difficulty 
^r uses less tnan optical strategies for designing and deb^ugging his 
pro^glTHnr, * t he i'nst rue toT^ interrupts and"^scril>es as appllcabTe ^ 
neuiristic to hin. The 'heuristics fre explicit, but couched in info^s^l 
speech. For exa::ple, if, a progras interftied to draw soc^e figure fails 

because- the student's design does njot tak^ into account an interaction 

^ * » 

between two procedures, he sight be told *'look carefully at -the position 

and orientation of the pen between^ ^he pirocedures that draw th^ p^rts of 

the figure that are incorrect^*^ Tnere are aeveral coGDeats-to be sade 

\ ' , , • * 

about this oethod. C^e^rly, it is not. a cost-effective .approach to 

s ' ' . ^ ^ 

large-scale instruction; however^ Papert has been concerned with gaining? 

, • ' <^ * • . . , 

initial Acceptances for its principles. with th^ idea that iEplesentatian* 

proble^xs can be resol'Jed su^^treatly. Second, arithou^h the students 

learcF^ an expUcit f'ornalisa (LOCO) for representing procedural 

V 



knowledge, the heuristics "^caselves *are expr^sed in natural laaguage. 



Finally, the interrelation^ aacmg the individual heuri^icj^^thin an 



/ 



enccapassing design and debugging strategy are not explicitly described 
to the studentH \ ^ 

V Carr and Goidsteln (1977) at. HIT have described a coeputer-based 
' sysce= called WUSOR-Il chat refioes Tapdrc's eethod of reactive tea^hinR 

of heuristics and exei^lifies how, it can be isade isore cost-eJfective by 
^ autocating. the TH)nicoring of tbe student, WUSOR-II is Wl^ STQupd a 
gaoe called Wuiq>us, a versiOQ of Fne^eus and the Mioptaur,' which 
reauires a. f undanental deductive prob^i^#lving strat^ £jor optisal ' 
play, rne player is placed-socewhg^ in a Ure of cavers.'xoid the' nanes 
of ehe n*»ighboring caves, and warped If certaiiT dangers are present in • 
tnose csves, alcnough the exact location of the danger is not specified 

, A a. ■ ' 



He ttien selects a cave to oove to. 'Ms goal is to find a'l/d slay the 
•» Wuspus by shooting an arrbw into the Cave where it is lurUag ^fote .it 

slab's bis. The reasoRirig involved .is fg^rly sicpl^ for exa^le, if a S 
cave nas a waTning and all but one of its neighbors are known to he< ' 
^^''^>'>'^^« danger is in the resaining neighbor. ^'Sote tnat this type 

re3s6.ftiag resecbles that required, in ti^ubleshooting/debugging tk. ' 

. ' ■ ^ localize a Tault given a set of observations. Trie optisal strategy fdr • r 

, ' ^ ' -/ , 

"Selecting a Eove is to deteraine the safest 'neighbor as deducetf^f roE t^l^ 

.nistory of warnings. ' ' • _ * ^ 

WUSOS-II incorporates dn expert 'Donitoring procedure, ^cause ' 
Che problec is well-structured, it was possible to iapletaenta ' 
' coi^utational oodel.for playing the optiaal strategy. The raonijor juses 
^ ^ chis-sodel to evaluate"the student's Dore. ^WUSOg-II incorporates a ' ^ ' 
■sopbiscica tea. pedagogical strategy to detersloe tdjen it is appropriate 
for the aonitor to interrupt play and describe a heuristic that ' • 

». _^ • generates *a better sove than Wie sttident had just selecjted. One of the 
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principles is to interrupt only vhen the student ^^s -consisfently failed 

^to sake Esoves that could be Ijsprove'd on by a particular he<!^ia£ic| that 

is, do not interrupt if the student fail^ tfo use a heuristic once vhen 

it is appropriate, vhen you have see^i hie use appropriately betore. * * 

Anothen principle is, based on a representation of . the intert.eia|ionships 

aaong the heuristics w^iich Carr add Goidste'in caira ' syllalyus : A ^ 
* * 'I ^ ^ 

•heuristic is nc^ Dentiooed unless the be^istics prior to it (e.R., use 

t 

of double evidence /epends on use of single evidence) is the syllabus 
are infarred to be' learned fron the Doves the sti^ient has oadei ' The ' \ 
teacning setnod Itself "(Figure 14) consists of , articulating the faulty, 
logic of the student's sove, the detailed logic for generating & better 
DOVe, and finally a general description of the heuristic used to ' 
generate tha.t oove* ^ . ' 

, WU^UR-Il is a noteworthy elaboration on Fapert's sethod 
explicitly p^efeenting probles-solvlngj heuristics. However, i^s 
capabilities are highly dependent on.the sicpHcity of.the problen 
donaln in whl.ch the heuristics ace taught.. The heuristics theaselves^ 
are part of a genial, deductive profc^^js-solving strategy that is 
applicable in nany problea jloaains, -including troubleshooting/debugging. 
An unanswered question is 'whether students tjho lea'rn g*eoeral heuristics 
in such a. "toy" donai/i can. in .fact transfer che= to a "real-world" ' 
donain and incorporate thea in oore c'onprehens^ve . strategies. A facto/ 
that alght a/fec.t (their sofecess is whether they have intJrrelated Ihe 
heuristics, Aey have had descrilted to tfiec^ separately across different 
probleo-solving episodes into an overall strategy. [ - ^ . « " 

The alternative to teaching heuristics reaotively one-by-one Is 
to introduce thea to the student according t<S a prior plan so' that they 
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'Ira, it isn't nece^arit to take such, large risjcs isith jdts. , > 
' ^ 'be nexi t*o a pit becanse f.it a ',r. f t there. B.nr.. 

_ cavey 15. 2 g^d m conl^U, „ ^t. tmt ha , e zaTelu ^i.ft.rf )>n». 4.^ 'r;;^.. 
wcaw tAot one-o^o»e3 2 ojid 14 coHains a pit, » 
£iAet^.e cat>e /5 rm.f, /.p „ezt to a Ht A.^ .tr^e f^f^ ^rc/t tfteCe. 
■ gence, one of caves 0, 4 and 14 contains a H t, but me hauj safelrt v'isit^a 
ca^e_4^ This aeans that one of capes- 0 and 14 co'ntal^ a 'pit. 

This is multiple evidence of e pit in cave 14 which mAes, it probable tijat 
oouc 14 contains a pit. It is le^ Ukelff that cape 0 contains a pit., -ffe/rce. 
Ira, we might oant to explore ca6e 0 instead. ' ^ * ' 



..'.gure ... Dialcg vitr. tte WUSOR-II systfc ilisistrating en att^^ 
* ^-^A^v .^j, 3 v^,^-.ie for ^plying niltipie. " " 

\ • I e-zlaecce In deducti^j. Frssj Carr afcd GojUoste^n, "SCT?, 
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are available to hiln whrenever be is ready, to use them, George Polya's 

book, "How to Solve It" (1957) is most often cited as the first Atecjpt 

'^to teach a problem^solving strategy directly by a text. A mathematician 

and teacher, he had observed basic similarities in the methods used by 

expert- p.rab^em solvers to solve mathematical proof problems. If these 

oethods could be described, he concluded^, they coiiW be taught to 

students, thereby saving the students the years it would take xhem to 

discover the^ethods on their otq^. Indeed, he 'felt some students never 

discovered these principles simply by working on exercises >y 

themselves, figure ij sucsiarizes the four stages of Polya's strategy 

-and the heWistics applicable at each stage.' Polya's. work though it is 

now recognized as' a precursor to inf orsJat ion-prof essing analyses of 

problem solving, has never had an impact >on practical -instruction in 

mathematics (Schoenfeld, i977a). The difficulty seems t;o be that people 

reading tne text mfay tmderstand \he strategy and heuristics but , when 

faced with a particular problfes^ have difficulty determining the 

particular Keutistic that "unlocks" that problea; that .is, while Polya's 

/ ' ' ^ 

descriptions are perhaps accurate, -^e way -in which they are presented 

^ in his book does npt enable ooSt readers to^adopt them as prescriptions. 

Wayne Hickelgren, an informations-processing psychologist, ^as 

^ authored a more^recent book, "How to Solve Problems" (1974), Wiich is 

Similar *to Polya's, but incorporates information-|5rocessin^- formalisms 

for describing ^.roblei structurj&s and probiem-sSoLving processes and- a 

presentation intended To teach the readfer how to recognize when 

' particular^st'rategies and tieuristlcs are applicable. Wickelgreri also 

does riot restrict himself to oatfiematics problems, but addresses a more ' 

general taxonomy of problem ty^ies. The probiem-solviog taethods-he 
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ftm. 

Yos lure to undmtand 
. C2te piobtezzi. 



SccopdL 

Find iht eonoectioQ Between 
wj]^ dau and the onknowiu 
You may be oblr^ 
0 cotxxlder snxtluty problems 
i{ an frrmifdute OKiaeaioa 
cwoot be found 
You ibould obcam vrtntuaMj" 
^ a plan of the yslutloos^ 



J IWOEMTANDINC TIIE PROBLEM ' 
FKA^e £r rte unknown} What are the dataf What tr the crindiiionf 
U h pouibfr to atbfy Ac condition) Is the comlitioa sufBcient to 
aetcnoine the unknown? Of b U insuffidcai? Or tcdundaoi? Or 
contradictor;? 

f^raif a figure Introduce suitable notatioa. 

Sqazale.tHc mbia partt^ the condition. Caa pu write ibeta down? 

» - ^ 

. DEVISING A PIAN 

Hate TOtt seen ft bcibrc? Or hare 70U seen the same problem b a 
xl%htl74iHercnt torm? ^ - 

l>o 70U Allow a relattd prebUm} Do pu know a iheorcta ^ could 
be ukfttU 

rooA tff tmknotml And try to think o£ a fiaultar problea harls* 
-the same or a similar unknown. . 
' Hertisa problem relate^ to yours and tolvtd before^ Could you me itf 

Could 70U use its result? Could 70U use its method? Should you intro- 

ducc some auxiliary element in order to osakc iu use possible? 
.Could you restate the problcai? Could you restate k idJH dillcready? 

Xio back to deBnitloos. 



If ywi cannot solve the proposed problem try. to solvfe Bm some related 
■proWem. Could yoa Ivrut^nc a more accessible rcbted problem? A 
moTtgcnml problem? A more special problem? Aif analo^us probl^ 
Could you njlve a prt of the problem? Recp-only a part of the condi-. 
toon, drop the other part; how far b the nnknown then detcnnhied. 
bow can u vary? Could you derive someiMnj; useful from the data? 
Could you think of other dau appropriate to dctmninc the unknown? " 
Could you change the* unknown or the data, or'both if necessary* So 
that the new unknoWn and the new data are nearer to each other? 
l>id you use all the data? Did you uk the whole condition? Have po 
taken into account all essential notions involved in the problem? * 



r 



Umd^ 



CARRYING GUI' TirE PLAN 

Cairyli^out'yoor plan |^he solwfon. cheek each step. Can yoa jee 
Carry out joux plan, oeazlj ^ the «ep u eorrectJ Canyoo prore ihat »t i» correct? 



. lOOKINp BAClt 

rotmh. Can you'^'ft'^'w/a/t' Cart jrou check ^e argument? ' " " 
ZJumint ihe lolutlon obtained. ^T*"* ^"'^"^ »ShI« diffcrentfj^ Can jqu tee it at a ({bnce? 

Can )Fou ujc the rault/dr tlie method, for join^ihcr prohWn? 



•"igure 15. 



Pciya's heuristics for proble*^ solving. Frczn Pcly/, 1957 , 
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considers include drawing inferences, classificatory trial and error, 
using an evaluation function to choose an action -(hill climbing) 
defining subgoals, deriving a contradiction, working^backw^rds, f rom the 
goal, and recognizing the relations between problems ♦ His approach is 
to describe in^g^aral terras, using formal representations when ^these 
exist, a problem tyfe and the applicable proljlecv-solvini^ toethoa and* then 
give a series- of examples which illustrate that method. The examples^ 
are most often puzzles- or mathematacai problems that require^a miniaal 
oackground, Ine solution to ^ach ex^n^le Is presested in steps the 
reader is instructed to attempt the solution according to the methcjd he 
has just studied before he reads each step. The text for each stepk 
describes a^ heuristic to.be applied at that point, allowing the reader 
to assess the heuristic be ^sed or to continue on if-he is stuck. 

> Wickelgren presents as comprehensive catalogue of prbbLem typea 
and me.thods j^s Qne could hope for'^iven the present understanding of 
problem solving. Two comments about the learnability of this 
information can be made. First, it is exemplified largely with toy 
pro±>J|/ems> a feature necessitated by the»^fact that the bookr i^^tended 
f6r a general audience and not as a text for student entering a specific 
discipline. Second^, although each method is described very thoroughly, 
they are not explicitly^ interrelated. ^Thtis, it still could be difficult 
to (Jetermine which method applies when a problem is^I>^e^1^rtted outside* ^ 
t-he 'context of a chapter descrlblng*aQ applicable method. 



Alan Scho€^nfeld, a mathematics inBisi;uctor ) has described in an 

unpublishe4 report /1977a) a method for t^achimi problem^olving 

r • * 

heuristics for mathematical'proof that buil'd^ upon th*e work of Polya-.and 
Wickelgren and that he has evaluated in a real instructional set^ng.*- 



He states that for a'student^to use a heuristic he must not only. 

» * * 

itoderstand the procedure it 'specif iesL» but aiso .understand the subject 

in which he is ti0 use 'it; 'and .recognize the situations in which it can }ft 
• « - jf " • # » 

.used/ The inbov^ion in Schoenfeld's met^qd is the explicit • 

ar&culatloh of :^«^at he calis^a managerial^ strategy , a prescriptive 

niodel of the relationships between individual heuristics. The 

^minagerlal strategy is ^ught to the student as^a dev^ice. for Bonitor'ing 

/his pr1>gress t'hrough the ^olutiod to a prqbles and » thereby focusing his 

attention op the subset of t^ hffristics h? knows thkt say\%e relevant: 

\ ft • 
at eac]^\point- Figures 16 and 17 are the heuristflcs andTsanagerlal ' 

strategy Schosnfelcf used in a small course he taught.' His tentative 

conclusion based on ao informal study of the soluttans generated by ' 

students on examination problems was that the students did develop- a 

better ability to select appropriate proof methods relative to sttidents 

in st^dard courses of joatKematfcs instruction. ' > 

Schoen^^d^ in a second unpublished r€>ort 1977b- descril^ 

-another small/ but ipore fonaal study ^aluating his method for teaching 

• ¥ 

heuristics/ this time for calculiis probleiss Involving indeflolte 

^ . — • . ^ - ■ * ^ ~~ 

integration. Fewer heuristics ^d a *aore limited faanagerial strategy 

* ' J. ^ 

^ - . 

are involved for this domain- Schoenfeld developed a brief t*>xt 

describing these and illustrating their application*, * The text was i^lven 

to half of a calculus class four days prior to an exaainaticfil - The 

examination involved nine probljeast seven- of ^ich cou^d be solved by 

the methods 'covered in Scbbenf eld's text, ; The^students who received .the 

text outscored those who did not on six of the seven problem > irtille the 

two groups 'did qot ^iffer on the othfer two yfobi^. ^rtheraore, the 

asCudents were asked to record the Uag th^y spent studying for the * 



ANALYSIS" . • 

1) DRAW A DIA6RAH if at all possible. 

« » ' 

2) ' EXAHIKE SPECIAl, CASES: * * ' .' 

^ a) Choose special values to"exe:?li^ the problea and get a 
■feel" for it. 

, b) Examine licriting cases to explore .the range of possibilities. . 

c) Set any integer 4>araaeters equal to 1, 2, S^.,., in sequence, 
and tooj; for an i^uctive pattern, 

3^ TRY TO SIKPLIFY THE PR03LEH by , f ' * 

a) e)q)loiting sysiaetry, or • * , 

b) "Kithoat Loss of Senerality" argraents (including scaling) 

EXPLORATION • ^ 

1) CKtSIOER ESSEHTIALLY EQUIVAtEKT PROSLEK: - 

a) Replacing conditions ^uivalent ones. ^ ' ' 

b) Re-d^ining the elements of the problem in different ways* 

c) Introduce au^iiliaiy el^sents.^ 

d) Re-forailate the problea by • - ^ 

i) change of perspective or irt^teti on - 

if) consideMng argi^eht by contradiction or contrapositlve 

iii)-/^susrfng you have i solution, and^detersining its 
properties 

2) COHSIDER SLIGHTLY HOOIFIEO PftOBLEKS: v 

a) Choose subgoals {obtain partial ftilfillsjent of the anditiote) 

b) ' Relax a condition and then try to re-i^$e it, 

c) Oe«)e^b the doesain of the problem and work on It case by 
/ case. 



Figure 16. Schoenfeld^s heuristics for-&olving matbematiisal 
I proof problems. FrcG-Scfaoenfeld, l^TTa. 
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(Figure 16 6<5ntinvie<3) 



EXPLORATION (contfrnied) 



•3}- CONSIDER BfaJAOLY HOOineo'mBUHSj 

^ a) Construct an analo5W& problea with 'fofcr variables. , ' 

b) Hold all but one variable fixed to <tetereine that variable's' 
•fepact. ' 

' • • • ,. 

c) Try to esEplolt any tested problej ^ith have sieilar • 

i) fora . > ^ , ■ 

11) 'givens" . * ♦ . 

ill) conclusions. . ^ ^ 

ResesSjer: dealing with easier reJated probless, you should 
• try to exploit both the RESULT and the mm OF SOLtrrsCK en the 
given problea. " , " 

VERIFYING YOUR SOLUTrOH ' .\ 

1) DOES YOUR SOLUTION PASS THESE SPECIFIC TESTSj, 

a) Does it use all the pertinent <fata? 

b> Does 1t amfoTB to reasonable estltates or predlctlSns?- 

Does it withstand tests of sysnatry, dlBenston ana1^is,.<r 
scaling? 

.2) DOES IT PASS T^^E fiDfERAL TESTS? ' 
aj" ten it be obtained differently? 
\) 6an It be substantUted by special cases? , . 

c) Can It be miffced to ioKwn* results? ' ' ^ ■ 

d) Can it be used to generate something you know? . * 
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ScHEHATtC OurmWE OF THg PrOBLEM-SoLVIHS STRATEGY 



Siven Probl 



ARALYSIS 



Understandfpg the Stataawt 
Siaplifylng The ProM« 
^ ;^ReforaiUtfng the Problei 



-Vseful ¥ormlstion_ 
Access to Principles 
. ted Hechanisss 



ted Pf^li 



Related 
• or. 
Ket Inforsstioa 



0 



Structuring the Argiewrt 
Hierarchical Oe<2o^)os4tion: 
global to Specific 



» Hilar- 

. OiffiojUies 
-> 9 > ■ >■ 



Bajor ' 
/Difficulties 



Schaptic Solution 



;6 



Step-by-Step Execution 
Local Verification 



Tentative Solution! 



EXPLORATIOS 



Essentially Equivalent 
fVoblcfis 

Slightly ffediffed " 
Probles 

Broadly Hodiffed^ 
Frobleas 



/ 



VERIFICAnOS 




Specific- Tests 
General Tests 



* VeHfied Solution 



Figure 17. Schecsatic representatiCTi df Schopnfeld's manageilal 
strategy fsr tsa tbea atlcal p|x>blea solving. Frcm . 
Schoenfeld, 1977a. . .' . • 
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exaalnation and those who studied with the text spent less tise on the 
average. ^ 

Schoenfeld's results, though based on a United saaple, do 
suggest that h^ufistics can be taught directly to advantage, provided 
- they are taught i^ the context of the dbmaXa in which they" will be used 
^ subsequentlr aAd they are explicitly interrelated within a \1Vger 

stflit^ay that predicates when each is applicable. In .th6' n^t section, 
we present a study that investigated whether^ a direct presentation oi 
heuristics can be used to teach inexperienced pi^raiisefs how to debug. 




IV. Directly teaching debugging heuristics; an experlateataL stud 



9- 



Raciooale 



IxK exaalniqg chronologies^ of debugginj^ behavior we foxmd that 

the difficulties of inexperienced progracsers are due as cuch to their 

lack^f a raciooal general strategy as to their unf aniliarity with the ^ 

declarative and procedural knowledge needed to understand programs and 

CO op^aie in a specif ic prograis^ing environment. In this section, we 

« 

discuss an experiment we conSucted, in which we attenpted to teach 

directly to inexperienced progracsers a f ew'^euristics that are part of 

% 

a useful (chough possibly conservative) debugging strategy. Tne 

Aperlsent was incended oore to be an exploration of -^thodology, than a 

definitive test of whether it is- worthwhile to teach representations of 

procedural knowledge directly. At the outset, Ufeltations on our access 

to subjects over an extended period preclu4ed any a'cteapt to teach 'a 

coDplete debugging strategy, or eve'n to teach part of a strategy 

thoroughly in a natural instructional situation-. Instead, a brief 

* . i 

tutorial fex.t was developed to preseij^-a few relevant heuristics and * 

subjects ^studied it only briefly in an experisental setting prior to 

^ » **' ' 

attenpting a few test problecis. Thus, we -knew that whatever tl/e results 

of the instructional treatsent, the adequacy of the peda^c^ used to ^ / 

coasunicate the heuristids' could be questioteed. Honeth^ess, for a 

first attespt to teach debugging heuristics it was^t unreasonable to 

test a ainiaal instructional aetfaod. Possibly^ the results vould 

indicate that sere Identification of generai^debuggiag heuristics is ? ' 

sufficient to fodtfy, tl^e* behavior if inexperi^iced prograaaer* (Wg", if 



they already^ "knew" the heuristics, but. needed an external cue to sake ' 
thea Bore readily accessible when needed). In this case; the costs 
developing -.Bore substantial, but unBece^agr* instructional aethodology 
could be avplded. 

The overall plan of^the e^erlsent was to co^are the -beji^vior 

'of two grpups of in«perienced programmers oh debugging problees, c^e of 

the groups studying and referring to thp tutorial and the other 

receiving only sone unassisted practice in debugging* Data analysis was 

to be exploratory, vfth a goal of identifying aeasures that could 

* . \ 
indicate ^th'e rrole of the debuggidg heuristics in subjects' prx^bLea 

solving. • I ' 

Debuggin;^ tutorial * ' 

The debugg^ tutorial we created presents ^Ight "guidelines" 
that are part of a general debugging strategy, "jollowlng the guidelines 
will not always lead to the sost , efficient debugging but for^an' 

inexperienced prograis^r without taich speci/ic debugging "knowledge they 

. - ' 

will tend to r^uce false starts ahd to 6elp determine a course or 
action irfiea he is "stuck." The guidelines can be seen as. elerents of 
three ^ncoapassing heuristics for (1) testing a"^ prograa sufficiently to 
detect errors, (2) generating a thorough characterization of an error's 

9 • ' 

Schoenfelrd's r-esulfrs- on teaching heuristics for Ba%heQatical.- 
proof problenss (described in/^he previous section) becaae available .only 
after the experisent describ^^ here was underway. In any case, there is 
a basic difference betweeiy^euristics for proof and integration pfoblgis 
and those for debugging. ,4n the proof probleas, -a single applicable^ 
heuristic sust be^eiect^j the subject's sain problea %^ recognizing 
the features of a jrroMea that make a specific heuristic applicable. '.In 
debugging, the us* of eereral b«ur4.8tics aust be coordinated at severalr 
points in everyi pjroblea; t;he sain problea in debugging is.reaeisfaerihg.tk 
us.e all of the heucfstlcs. Of course^ in both cases the hduristics-Aust 
be used appropriately. • ' , .«•• 



zaanifestation, and (3) backtracking ,f roa unsuccessful repairs, A 

s uTwa ry of the guidelines from the tutorial is shown in Figure 18. The ' * ^ 

nuiier next to eac|3 guideline indicates which of the three heuristics it 

* • - • 

is part of. Tke heuristics were decoq[posed into separate guidelines to ^ 

facilitate tfieir conprebension. The guidelines are* shown i"n Figure 18 

in the order in which the tutorial introduces then. This order reflects 

that in which the guidelines are applicable '^urin^ each iteration (t>r ■ 

recursion) of t1he general debugging st'rategy that was described fh 

Section II. - ' " ^ ^ 

The eight guidelines were fonsulated ^^o correct the ©ost 

frequently observed shortcoaings we had previously identified in the 

debugging behavidr of inexperienced programsers. Ali of these ' ^ 

« 

guidelines J except perhaps for those concerned with»backcracking,. have . 
straightforward nappings onto other troubleshooting situations, like 

> 

electronic and csechanical maintenance and repair. For exac^le, varying 

s , * " 

'a progran's inputs is analogous to varying the inputs and external 
contwrols of electrorillc and cechanicai devices. '-^ 

The tutorial (Appendix A) is a rather niniaal piece of pedagogy, 
in a linear narrative sode, it introduces each guideline, giving a % 
rationale for its use and a specific debugging scenario that illustrates 
its successful application* The exaoples are intended to deisonstrate 

it is appropriate to apply the guidelines: having 4>robleQ-8olving 
heuristics' available is of little ^use -if one does 'not know the 
circuastances under which they should* be 'applied* The exasfile programs ^ 
were taken with^ slight modification f roa the prograiiJfiing chronologies we 
had exacdned earlier. The narrative for the exagplea was developed in» 
part froa the written coaaentarles we had collected froa the 
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\ .<i) 



(1 ) 



t27 



(2) 



< 1 > 



(3) 



TESTING THE PROGRAM 

TEST THE PRQGRAM. WITH ALL POSSIBLE TYPES OF INPUT FOR WHICH 

IT IS DES-IGNED 

' TEST the" PROGRAM WITH THE^(cTB01E, VALUES THAT THE 
INPUT CAN HAVE ' . 

CHARACTERIZING THE ^ERROR 

CHARACTERIZE THE WAY THE ERROR (S)- SHOWS UP IN TERMS OF THE 
INPUT AND OUTPUT #• 



(2)- EVEN I-F A PROGRAM 'IS SHORT AND E^k TO TRACE BY HAND, YOU 
. SHOULD FIRST RUN THE PROGRAM. (UtoR MESSAGES. AS WELL AS 
. A CHARACTER I 2AT ION OF THE ERROR IN TERMS OF INPUT AND OUTPUT, 
CAN BE VERY. HELPFUL IN FINDING AN ERROR ) • ' 

SOMETIMES A PROGRAM GIVES; TftE CORRECT OUTPUT FOR SOME INPUTS 
BUT NOT FOR OTHERS. WHEN THIS HAPPENS YOUU SHOULD EXAMINE THE 
DIFFERENCE'S) BETWEEN THE INPUTS FOR WHICH THE PROGRAM WORKS 
AND THE" ONES FOR WHICH IT FAILS. 

AFTER A change/ RETEST THE PRQGRAM USING ALL POSSIBLE TYPES 
OF INPUT FOR WHICH THE PROGRAM WAS DESIGNED. 

IF YOU MAKE Jft CHANGE TO A . PROGRAM,- AND IT STILL GIVES THE 
SAME ERRjONEOUS OUTPUTi RESTORE THE PROGRAM TO ITS fejATE - ■ 
BEFORE\THE CHANGE. ' YOU HAVEN'T FOUND THE ' ERROR (8) IN THE 
PROGRA*, ANa YOU MAY HAVE' INTRODUCED A NEW ERROg, 

<3) IF YOU [make 'A CHANGE TO A~^ PROGRAM, /^D THE OUfPUT fS STILL 

WRONG: t iF THE CHANGE CORRECTS ONE PART OF THE PROGRAM .(e^ g. , 
. , one parV of. the output), THEN LEAVE THE CHAte JN THE PgOGRAM. 

• Figure IB.^ List of 'the Oebugging guidelines presenled in the debugging 
tutorial. Hufflbers in parentteses indicate grouping of ^ 
guidelines into three enccoroassing heuristics for testing, 
characterizing, and backtracking. , - , " 
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inexperienced prograEsaers;;^© had tried to debug thea. The eoimsentaries 

contained instanpes^ of'both productive and non-productive reasoning 

useful for illustrating how the guidelines could help during- debugging. 

By using exacq>le8 Vith errors* and problem-solving introspections ^ 

actually produced by inexijerienced programmers, we hoped to create a ^ 

text consistent with tte experience of subjects w^ would eizg>loy. Other 

than indentation* and emphasis, the tutorial makes no Ose *of text 

engineering techniques like hierarchical outlining or Systematic review 

which might improve cdaprebetision. In fact, th^ tutorial assumes a high 

level of literacy and motivation. These were characteristics we 

* 

expected of the undergraduate who would participatfe ^as subjects and we, 

cho^e to make the text consistent w^h their aptitudes/ We did create a 

♦ 

brief opeiibook test. (Ap|>endix B) to accompany the tutorial so that th^se 
subjects could monitor their comprehension and determine their own# 
review strategy. ^ 

Procedure ' » • 

*^ub1ects> The .subjects were twelve paid volunteers ftom a group 

of 28>Btudents who completed fifteen hours of curriculum in the BIP 

• ' * 
course in the weeks prior to the experiaent. Thiis', at the time of i:he 

X ' % 

experimeqt, each subject had written several dozen short programs within 
BIP, but had no other programming and debugging experience. The ^ 
subjects^were recruited .approximat^y halfway through th6ir . - 

participation in BIP^ 

» ' ' . • 

Pridr to beginning BL^, each subject had been pretested with the 

Computet 'Programmer Aptitude Battery (Pelorao, 1964)^ On the basis^of 

their scores, t|ie twelve suBjects were divided int^ two "matched" groups 
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.of Six subjects each.' Ihis waa done in an attempt to contro'l for ' 
pte-existing differences that night Interact with, the experimental 
instructional treatment^P * ' , .. 

.Experimental environaeptL. Ail experimental sessions were 
conducted in the same setting tn which subjects h^d' worked wiirh'siP. 
All experimental test exercises were conducted using BlP's prograaaiflg 
facilities (of course, those facilities specific to the- BlP'curriculum-- 
e.g., HINT— were inoperative for the test exercises). Two^?CRT^ 
terminals were available, allowing either one or two subjects (always 
from the same experimental condition^to be scheduled' for experimental 
sessions* 

An experimepter was available throughout the sessions to help 
with procedural problems (e.g., loading' exercises into the subject's • 
program space in BIP and recovering from system crashes) and to list 
hardcopy of tesC programs for 'subjects upon their request. " . 

' Method . Each-subject participated in three. sessions. Session 'I 
was the (different) tfeatment/testicfg session for the experimental * 
(TUTORIALi and control (NO-TUTORIAL) groups. Sessions 2 and 3 were Test 
sessioqs identical for both groups. 

^ Session 1 foT the TUTORIAL subjects began with aAtext 
introducing the general logic of trdubleshooting/debu^ing (Appendix C). 
The subject was then given the tutorial text pifesenting the eight ' 



'iO , ■ ' ■ 

Scores rpflecting ability after BIP would have been 
preferable, but cotjld not be used because time constraints requirfid tfcat 
each subject >«gln Ithe experiment as sqon as he cottfpl^ted/hls 15 hours 
in BIP. Thus, asftighoent to tteatment groups had Co beiaade 'while 
subjects were stilliin BIP. Subjects did* take a prograaming posttest 
after completing Bl^, and before: participating in the experiment. 
Subsequent analysis ksee RcBults below) indicate that the two groups of 
aubjects differed «atkedly with respect to the posttest .'scbres. 

1 • . " ' 



guidelities. to study for'bne-half Kou'r. ' After study, t^e. open-book quiz 

was gAen apd the subject revifewed Jthe text as necessary to compile the 

, quiz. The subj^ect then' moyed,to the terminal to work a debugging - 

problem. He was given (in his Jprograo s-pace') a prograa and" a 'written 

description of its, intended function. He was toi5 tbat-^hei'r was ' 

something wrong with the program knd that his task was 'to change it so 

that it worked according to the descrlptiolt. ^th^ J.n^tructions\ 

euHJhasizfed that trhe necessary changes were minor and that hVwae nor to 

- . ,\ , ' ~ ." 

write his own program to satisfy the descriptions A time limit of 

• ' ^ 

one-half hour was, imposed for this debugging exercise; 

The test progrkm was anvatypical solution to a taak In the BIP 
^curriculum ca|^ed CHANGER. All of the subjects had worked on this -task, 
but- had used algorithms different from the'one in the exercise. CHANGER 
i% supposed to ask.^the uses \^o^^ purchase pcice less, than one dollar 
and print the amount of change and the' list of coins needed to make that 
amount' of change most efficiently. The bug in' the test wSrcise-: 
manifested itself as an incprrect list of coins whenever two dUfes were 
required as part of the answer {Appendix .D).' ' . ' ' ^ 

Session 1 for the NO-TUTOftlAL subjects, began with a brief 

description of tfteir t^k-. The subject ^th^nvas^gljr^Q one-half hour- to 

work at the terminal on the first of two debugging exercises. Again. ' 

the exercise involved a malfunctioning- progtao, ^description of -its - " 

intended function, and instructions to seek a oinlaal repair. The 

program-was on^-of those used as an example in the tutorial "to 

, . ^. • • 

illustrate the use of the' guidelines ^or debugging* It 'was- chdBeo as an 

exercise fbr the NO-TUTOR lAt group in order^to mlnialze. diff erem;S8\ln . 

knowledge about specific prograas and bugs. Uie ^second debugging. 



^xerciae, given 'during the: second cme-^^f hour of Session li*waus the 
(iHANQER ,program given to the 'TUTORIAL sulj^ects in the secdnd-half of 
their first sessitsp^ 

•Testing in Sessions 2 and 3 was identical for ^oth^ groups, 
except thart subjects in tKe ' TUTORIAL group, could refer\o the tutorial 



text -as they pleased.' The exercise given .in Session 2 was to wVite a 
program DRILL (Appendi3^ E) • DRILL, a prdgram to provide 



drill-and-practice in addition anld ;6ubtraction, was longe^^^fch^a 



aore extensive control structure lihan any proTgraa that 



t^^pjects 



were 



reqirired to write in BIP. The necessary control structure was such that 
the goidelinea fox program testing given* in the tutorial could 
T%a§cf^ly .be^ expected to facilitate its successful impjementat^dn,. ' Two 
houra were allowed to'v^rite the program. If a, subject completed the 
p^rogra%"within one and onfe-half hours, the experimenter tested it and,., 
informed the subject simply whether it did <^did not Satisfy the 
specifications. If it did not, the sub^fect was allowed the .remaining 
tim^ to complete (or debug) his program/ This was done to* provoke 
drugging in cases where a, subject had not been able to detect a bug in - 
his own prog^ram. ^ ^ • ' / ' 

The exercise given ia Session 3 was to. de^ug (under inst^ctlons 

lafentical to those for the debug^ng exercises of Session 1) a program • ^ 

■ . ' ' \^ i ' ^ > - 

ARITH-CALC which had been written and "bugged'' by one%f the tesearcli ' 

team (Appendix. E?^ MIT^^^ALC -was designed to ey^uate in Strict ^ * 

left-t:o-right;ot:^er strings^representtng numeric expressions input by ^ 

user* <BIP's dialect of BASIC has no ^lutoTaatic type-conversion 

mechani^.T'Again, the program was longer and more complicated than 

tiftyse required isy the BIP curriculim or used in the tutorial and . 
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Sessiorf l*^ In particular,' the algorithm almost certaiirly was^unfamiliar 
to $11 the subjects-^'^nd difficult for them to .trace aentally^ although^ 
"no specialized background knowledge is required ta gain an understanding 
r-^ , of ARITH-CALC and- its^ bug werl generated so that '•the tutorial 

g<iideline^ for characterising^ error 'T.n terms of input -outppt 
relationships would be relevant for efficient solution of the exercise* 
The bug does not manifest itself for evei^y input and the discreptocy in 
). the output value varies aa a function- of thfe arithmetic operations and 
/ their order in the Input string. Subjects w^re given one and one-half 7j 
hours to complete thie exercise. ^ ' * 

In all sessions, data op each subjectTs programming 'and 
debugging behavior was collectecL automatically (apdU invisibly to 
subjects) by-BIP's chronology facility. In addition, subjects .in the' 
experimental group were given a written questionnaire at ft'h^ -conclusion 
^^/-^of Session 3 designed to elicit their reactions to the tuto*^ial and the 
experiment (Appendix G). ' ' . * 

■ ' • * ' ^ . . c 

Results . ' • . • 

Out efforts at earlifer stages fhe research to derive* 
• » * # 

debugging grammars to describe-^IP chronology data had been 
^ unsuccessful. Therefore, we did aot "have available any cosprehenslve ^ 
mechanism for analyzing the chr'onoldgies collected in the experiment in 

order to describe differerieea between subjeSes' strategies. The type of 

■ ' ■ - .- • ^ 

\ ' analysis we conducted was thus anich Bore limitedcthan we desired. The 
present expejMjaeot was cotn^eifn^d specifically with -4<1) whether the 
behavior of subjects 'in the TUTORIAL group would refleqt* fheir attempted * 
^ . .' use of the guideliTjes given in th^, tutorial teSet and, (2) if so, th€i 
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extent to which the guidelines wer^ in fact acquired froa the *text ' 

rather than inferred frcra prior experience (as detenaii/ed by poaparl^on 

with the lO-TUTORIAL group),. The clironologies were analyzed to assign 

- ' ' * • _^ X ' ^ 

values to five relevant "peasures" for each exercise. 

9* 

(a) adequacy of solution , ' ^ 

(b) detection of bugs via program execution ^ • 
(as o^pJosed to taental analysis of th^ code) 

(c) characterization -of bugs. via extended program execution 
^ revealing input-output relationships 

(d) extended testing of atteapsed prograa repairs 

(e) , backtracking fyom unsuccessful repairs 



The Erasures represent the success of the attrf^ted solution aifd the 
extent and success with which^he heux^istics etfcoapassing the guidelines 
were applied. ^ ^ ' 

Each measure ^^as assigned a value meaning "done 
successfully" <k meaning ^either "not successful" for -a or ^'not 
attempted" for b-e* Measure b-e could also be scor^ as "0" ae^ing 
^"atf tempted, but with unsuccessf url results." For instance, a "0" value 
would he assigned to ffeasure b if a subject ran the program several , 



tiaes with different inputs, but failed to find inputs that caused the 

bug to taanlfest itself . ,ln addition, some a^asures could be scored **HA" 

©earing "not applicable"; fc^r exaaple^ if the sul^ject nevW attospted 

repairs, no ^core qould be assigned to iseasures" d and e on that 

exercise. ] - * 

^ > ^ . o • - ^ . . 

Determination of scores for aeasures b-je froa the chronologies " 

proved to be- a rather* coap lex ^judgeaent process. There are na. singular 

events in the chronology fo^ an eacerclse that determine unaabiguously 

t-he values of- these foiir aeasures. For exac^le, whether or not a 

subject actually characterized an error in tefms of t^e input-output 



- " s • * 

. relajtionships obtaifi^ ^>X/his execution of sl program can be determined 
• only by examining his subsequent repairs and the other events leading up 
to then. In a sense, v. the scorer had to try t'o simulate ^he reasoniag 
underlying the subject's actions and see if it waff consistent with a 
hypothesis that the^ error bad been charaSte^rj^zed in te^ms of 
input-output^ relationships. A further coii^licatlon is that an attempted 
solution n^y involve mo.re th^n one debugging cycle, ^ or episode* In 
these cases, scores fof.the oeatures^were determined^ by judgement of the 
predominant bel^v^or aclross the episodes. , ' ^ . 




^ . In order to tred^ce pa^^tr^ial 'bias in this subjective scoring 

^^^^^^^^^ \ • -« 

process, chronologies '(whiph contaiif repeated information identifying 

subjects) were scored primarily by a member of the research team not 

familiar with the assignment of subjects to groups. However, no data 

V 

was on the intra- aod inter^judge reliability of scoring fdr the results' 
to be presented* ^ • ^ * ^ 

Results will be presented here for the debugging^ exercises ' 



CHANGER ^nd ARITU-CALC attempted by both grdups in the^second half of 

Sesaion i and in Se^s^ion 3 respectively. , Behavior in the programming 

exercise DRILL given in Session 2 proved impossible to tecore with any 

degree of confidence because ol the great Variability w4th which 

subjects approached at ^ Some subjects in fact, never implemented 
~ * * <• * 

enough of the program to, execute it and examine any . output. Others 

projkiced, executable piece^^*?^a solution program, but showed widely* 

varying debugging behavior ^in different episodes within the exerciW 

Given these difficulties and the fact that, there were no differences in 

the mmber of correct (or 'almost correct solutions^—- aeasure a~ 

• *. , * 

between the TUTORIAL and NO-TWO£^IAL groups, it see^^^^pointless to 
' score the chronologies for DRILL with respect to li^sures b-e* . ^ ^' 

■ ' • ' • '■' v.-".' Ii2''- ■ ^' ' 



Tables 2a^an^"2b prtesent the five scores on CHANGER and a 

' J 

ARITH-CALp and also th^ BIP pretest and posttest Scores. for each subject 

in the TUTORIAL and NO.-TUTORIAL groiq>Ss Most striking is* the poor 

performance of SjUbjects in both groig>s as indicated in coHmn a* three 

laembers of the TUTORIAL group and two seabers of the HO-TUTORIAL grbup 

Solved neither o^ the probl'ess. In each group, exactly five exerc:j:ses 

were coapleted successfully. Thus,*^^fce instructional treatsent for* the 

TUTORIAL, grou^ does not seea to have tmpr^^ their debugging ability as 

. measured on two test exercises specifically f insulated to be sensitive ^ 

to that ^instruction^ Unfortunately, however, evpn If there w^ an 

/ / ' ' ' ' 

effect of the treatment, it "may have beea obliterateH by a difference 

betwecQ the ability *of t>he groups at the tiise they began ihe experiment. 

Recall tbzt groups were matched* using the BIP pretest scores. 

Inspection of the posttest score^s, a^jail^ble to us only after soae * ^ 

^subjects had beguS? the experiment, sho^s that a largTe difference i|i 

programming ability existed for the* two groups* By chance, the subjects 

assigned td^'the NO-TUTORIAL group had become auch better prograaaser^ on 

11 * - . ' ' * • 

the average. Thus, if the tutorial-did iaprove debugging ability, the 

only* effect ©ay have .been to cancel the iuitial difference between the 

. ■ » * > 

experimental anS control groups. * ^' *, 

' The small sample size precludes a meaningful statistical 
% ey^^uatian of the difference; however, in our experience, s^uch a l^rge 
dif f ecence in i$osfetest scores does^ have practical significance and 
correlates i^ith subjective it^resskons of programing saphistication** f 

12 ^ > ' 

Afl attempt was ilade to obtain "dljEference" scores for each 
subject in order to see 'if the TUTORIAL group showed a^larger " 
improvement in debugging. ability relative to their ability before ^ ^ 

studying the tutorial. BIP chronologies for the final few BIf tasks 
worked by each 's^ubject were examined^ However^ th^ variability , in these 
chronologies resemble%th^t found in the transcripts for the \ . 
expeifimental* DRILL exercise* Thus, .It w^s not possible to score the 
pre-®dperiaental debugging episodes with' anV confidence and thereby to 
pbtain the desired difference 8cbres« / ^ 
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HEP Test Scorea/anB Debugging Mgastires- 
* for ^UTOHLAL Grotg? 



Subject 



HEP 
pretest 




3iS 116 
129 

322 ' 131 

r 

333 ^ 79 
31,0 56 




Debugging Measures* 



ill 
8^ 
IBI 



\ Task 


a* 


b 




* d 


e 


-cWiger 


- 


+ 


+ 


EA 


+ 


Aritb-Calc 




+ 








Charter 




+ 




+ 


EA 


AritB-Celc 


+ 




+ 


+ 




Change 3f 




'+ 


+ 




+ 


Ailtli-Galc 


-I- " 


+ 






+ 


Changer 




0 


0 . 


EA 


HA 


*Arith-Calc 




-¥ 


0 


0 


+ 


dhanger 






0 


KA* 


EA 


Aiith-Calc 




+ 


0 


0 


+ 


Changer ^ 


+ 








EA 


Arith-Calc 




4- 


+ 


m 


0 



^ ~ — * 3 

*Key: Measixre definitions 
a Solved problen 

b -detecticm of bugs via p^^x^graa exec^^^^^ 

c cfaaracterizatiba of bugs *via exteiSSidprogrss executim 

d erfcended testing of atteigyted rq>airs ' ) 

e backt2:BcfcLn§ free vufeuccessml re^oalrs 

>\ Measure scores \ • ^ • * 

^ sxzccessAil V • 

not successfQl (a^ or not atteagjted (b-e) 
0 .att€sq?ted^ unsucc^ssftQly (4>*?) . 
9A not applicable in solutiOD coniSext 
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Table 2b 

HIP Test Scores and Debugging MsasxISres 
for BO-TOTORLAL (Jroirp 



'HI? 



HIP 



Subject pretesr posttest 



a * b c ' d 



e; 



311 

319 
326 
332 
337 
339 



113 

i30 

115 
y 

ivi. 



237 
1S6 



CTnanger 
Arith-Calc 

Changer, 
Arith-Calc 

Arith^^Calc 

Changer 
Arith-Calc 

Changer 
Arith-Calc 

Changer 
Ailth-felc 



+ 



+ 

4- 



+ 



+ 
+ 

+ 

0 
4 



0 
0 



SA 

EA 

/ 

0 
-I- 

HA 



^ek: de£Lniti<ms 

a soI%'ed problsn . ^ <^ 

b ^ detectlcaa of bu^ via^progrsa executi^ai 

c characterizatica of Inags ria extended pr^^ras '^cuticc 

d extended tpstiijg of ettez^rbed reprirs 

e backtracScLng free ttnsucc^ssful i^jsirs 

Measure scoi^s ^ * 

- ^ot st^<^sfta (a) ov not att^jrtied (b-ej » 
0 att^ipt^^ imsuot^fsftQly (b-e) 
-KA not .agpHcabie in solxEttcai c^mtext 
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The completion times for the correct soluticfos to the CHAHGER 

'and ARlTH-^ALC.ejeercise&.were ^iso exaalned to evaluate tlte hypothesis 

^ that the TUTORIAL. group -vould debug acre rapidly than the HO-TUTORL«. 

grotq). The' observed Bean coj^letion tioe, however^ was shorter for the 

NO-TUTORIAL' group, prisarily because of Subject 339. As indicated 

his posttest score in Table 1, this s^J^ct was the nost proficient 

.prograaser at the tioe of the expe5%ent. "^(^ was also unusually.' I 

. motivated, being one of the few students in BIP who had* generated his \ 
• * 

own prograEsing exercises tot^suppleisent BIP's curriculum.) He correctly 
debugged both zmm^ and ARITH-CALC in sho^t order, characteAzing, 
locating, and repairing the errors ^arencly bj^ analysis' of the progras 
code with littleiatfeation to tbe.data provided by prograa execution, 
rnus, the debu^^ng exercises, wfrtQhwere difficult for the aajority of' 
subjects, sees to have been tOQ-^aSyTo tax the ^iUty of Si^ect 339|. 
Cocsequently, the data do not indicate that the TUTORIAL group debugged^. 
"cH>re rapidly. 

Returning to. the aleasures in Table 1 for apparent use of the 
heuristics given in the tutorial, there is Wginal''evidence that -eve 
if the the TUTORIAL group did not sqlve aore problces (or solve 
Bore rapidly) than the SO-TUTORIAL group, they did afte^t tov^ply the 
gttidelinei fop testing <and debugging, lie coluais labeled'i-S *' ' 
correspond to the eeasures described earlier. Cojtim b indicates 
whet|»er prograa execution was atteepted and successfully caused error ^ 
Banifestatiojiiefore.the subject engaged inuother debugging activities. 
For TUTORIAL subjects, such detection was successfta in every case, 
except one where the prpgraa was executed several tises, but the input? 
used did not cause error aanlfestatlon. While 80-TUT(®IAL subjects also 
did so freqVi^tly, in 3-o£.tht 12 cases they did not. 



" • / ; ' . < 

* jfc' £ indicates 'cjiaracterization of erro/8'.by"progf^a ^ 

V . / executijto sufficient to elaborate a description of.tJie malfunction. - ' 

. TUTORIAL subjects attempted to <}o so in 11 of 12 cases, although in 4 of l 

those casesr the attempts vere Judged to be inade<juate| the' corresponding 
. - - results foe ihe KO-Tin:OSlAL group are 6 Vf 12, ulth 2 inadequate ' 

acteopts. - " * 

Colunns d and e are the measures of repair testing and - " ' 

backtracking fron unsuccessful repairs.'- Both groups show equivalent 
evidence for such behaviors. , 

ExaelDing oeasures b*^'just for the exercises that were 

cocpleted-succes^ly'^oeasure a), it is Interestinil toWe that for | ' 

the TUTORIAL group all of the guidelines were applied iD>each of the 

five cases. For the NO-TUTORIAL group, in 3 of the 5 porrect Bolutions, 

the bghavior prescribed by one or sore of the guidelines not 

obser-^ed. On the whole, ^ seeas that the subjects who studied the ' 

tutorial did try use the guidelines. However, the data from the 

NO-TUTORIAL groups does suggest that a aajority'of student prograstaers : ^ 

• 

' ^'i'Jb.tbe expsriefice level of our subjects have already induced aost of ' H 

. ■» • . . ' ■ : • . > 

' th^ guidelines (or sisilar heuristics). The differences "between the . « ■ 

group's are soall and allow no styong conclusions', She tutorial text say ^ 
sisply have served to aapltfy and organize parts of a strategy already 

• ft 

known to the subjects who studied it. . ^ 
The written cotaents obtained, f roa the TUTCJRIAL subjects at the ' 
** . • cotfclusion of -Session 3 provide soae help In deterHlfiing the effects of 

^ th<texiu»n their behavior. . Figure lists the sore iaf«iraative''' 
^ . r^apks 'that subjects aade to 'items 4-7 shown in Appendix G. The " . * 

y ^ ^ consents about ,the tuijoriai are positive for the oost part. With the - 



Do youTJave any 'suggestions (criticisms), in general, regarding the manner 
of presentation of the guidelines? 

32l»— Should have been.iaore tim to 'study them. 

^ . ,■ . •• 

W<iild iH.'bave been better if the guidelines had been g3.ven to' yew. before 
you finished tte HEP coarse? . • 

, 31c— Perhaps, better in the long v^. Actually ended up doii^g the things- in 
the guidelines as tiioe vent on. , Of ccnirse, having theii given to you 
right avayVis less tiise copFuzsing since you don^t have to grope around 
trying to decide vhat to do next. - * ^ ^ 

* " » 

— 1% have helped^ but ncase of the progracas in the cou^e veie that 
cGciplicated that.it vas .oecessaiy; and coHTt if it vas fairly obviois. 

222— Didnn ^ally i^d it in BIP itself 'except for coirolex prt^^g* 

32^" I cc^zld have studied^^t:^ at leisure and really leein^ them 

veil. 

— i^sn't tnat irach difference- for BIP ve didn^t have so mch 
' as to debug progi^ss* It vas pretty mch' follcv the exainples. 

Sot rkecessarily, these guidelfties are pretty basic things to do and 
self^discovery is probably as useful^ ? 

Dc ycu think it vaold be useful to have BIP introduce this material as part, 
the course? . . * ^ . ^ ^ 



316— Yea, ' " * 

317— Yes^ it^s good to kn6v. ' 

3^^ — Yes, before pre^erttation of cc^lsx problems. 

^ u 

321^ Yes. • ' ' 

333 — Yes, it does help a bit and might relieve the frustratica of not havl^ 
a program voiic and hot knoving hov to go about finding ^irtiat vas vrong. 

3^*0— Peitaps. ' ' • . . ' ' ^ 

- — „i ^ 

Other CQRsaents. i i * ^ 

* ' ' / 

3^— The last 3 sessions made del«ggii% seem a mich moii^rderly processor i#€#' 
' more manageable* 

Figure 19. Replies of subjects in the TOTORtAL groi^ to questions 

the post-experimental interviev (i^ppendix G). r 

■ . ^ ' 109 ^ , . . ' 

• ^ 118 ; 



exception of Subject 340, subjects thought that the guidelines were, 
valuable knowledge, alt hough- they were' not in agreement about how useful 
.they c'oua/d be for corapletiag tasks in the BIP curriculum. Several of • 
the subjects recognized that the guidelines are knowledge that they had 
or would have acquired i;idirectly through experience, but thought that 
the idea of teathirig such knowledge explicitly could be aore ^fficieni^. 

Th^ debriefing d&ta does point to the inadequacy of tttniaal 
instruction, such as our tutorial, for insuring that heuristics will be ' 
learned and used by students who need fhem. The ratings given by 
subjects on items 1 and 2 of the debriefing questionnaire suggest that 
(l> they did not find the tutorial especially useful for the test 
exercises they worked" in the experiment (five "3!"s and one "2"), and~^ 
(2) they thought they were fallowing the guidelines most, but not all,*— 
o.f^,the tlDe (four "3"'b and two "2"'s). It ijs very inte/estii^ to npte 
that the two "2'"s on item 2 came from Subjects 324 and 333, who had the 
lowest posttest. scores ^n the TUTORIAL' gtoup (Table 2a^. This. again 
suggests that ''the students who had the isost to gain from the guidelines 
could not, or would not use them consistently. These two subjects were 

% 

the only ones who reported referring back to the tutorial while they 
worked,' and 324 was the subject who remarked that he dtji not have enough 
time to learn the guidelines. The other TUTORIAL subjects seemed to 
know the guidelines, but failed either to use all of them as* regularly 
as they might have or to use them' appropriately lox the test exercises. 

* * 

Discussion and Gonclu8ioDB » G ; 

* ,^ • « ^ . - _ ^ ~ - 

The refults of t5e expertsent serv? 6o mual^ie. aat^odologlcal 
i8«i»8 xaore than to answer the question of whether it Is worthwhile to 



J 



teach debugging heuristics directly. Both the chronology data and 
subject's conufients hint that TUTORIAL subjects recognized the value of 
th6 g^uidelines and tried to Aise them, but provided no evidence that they 
becacie better at debugging pro^^rams* The comments are most encouraging, 
biit sQould be weighed cautiously, since the conditions of the experiment 
may w^ll have prompted the subjects to tell-tis what they thought we ' 

wanted to hear. 

f ' 

As noted earlier, we were- awara of some methodological problems 

at the outset of the experiment, and our subs€?|uent experience has- 

highlighted these and some other problems that •must be solved before a 

substantial evaluation of.^feaching troubleshooting/debugging strategies 

di rectly cail be conducted* 

One problem is developing a pedagogy for teaching heuristics — 

tpr teaching procedural rather than declarative knowledge. Although we 

could rationalize a first attempt involving tainimal instruction, we 

-anticipated that the limited study of the tutorial, isolated from other 

mstructidn in programming, would be insufficient for precisely those 



students who most Deeded to improve their debugging-*- the students who 
had as yet not induced a viable -strategy om their own. It is to be 
expected that meaningful learning^*&f complex knowledge requires 
considerable time' relative to the'learnlng that takes place in 
laboratory studies of learning. Qur situation of having limited accjess 
to student's time is, ofrcour^e, the rule rather than the exceptidn jin a 
basic* reseacph setting. There is ft '^Catch-22" of sorts in effect: it is 
difficult to persuade and possibly unethical to compel^ tuitions-paying 
students to participate in an ugvalidated,. Innovative instructional 
program, but one cannot provide the needed validation without testing a 
sufficiently large and representative first group of -students* • \ 

■ . , '""120 - ' ■ 
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It. is usually possible (as we did) to gain the cooperation of a 
small group of volunteers who* tend to. be either students having ' 

^difficulty and peeking any neans to improve themselves or students who 
are unusually -bright ^nd motivated.' These individuals are not ♦ 

' representative of the student populat^^fii. FurthenMre," small groups- of 
volunteers not allow for statistical tests of hypotheses which are 
needed to validate an instructional treatment. 

In sonfe cases, it is possible to ^ain actess to a large student 
population; for example, if the researcher or s^athetic colleagues 
teach a course into which the new saterial can be integrated, ' However, 
there are ethical issues that surrouhd the .coi^jalsory participation of 
tuition-paying students itj experimental cour4eBi that are extensions of 
spo.nsored research programs rather thkn pvodifcX^ of 33 instructor's 
initiative. I'f the effectiveness of fche/infefuction is very tentative, 

j-chen students should not be compelled \^ participate. If the 
effectiveness is highly probable (and the experlfflen\>iing^ conducted 
only to collect supporting data), then how can a control group- that 
receives less than the best instruction fo"r their time and tuition b€ 
justified? > • . . 

A second methodological problem we encountered Is to determine 
test exercises that will be sensitive to differences that might result 
from the^nstructlonal .treatments. The solutions to debugging exercises 
like those we jasei irequire general knowledge of a programming language 
(e.g., BASrc) and of a supporting computer system (e.g., BIP). In 
addition, idiosyncratic knowledge acquired from prior debugging may .be 
applicable to a solution. Therefore, test exercises Intended to 
indicate the role of general debugging heuristics caq neither be too 



eleowtary.nor ^oo,.advanced.- .If tljey are toa eieaentary ^and hence 
faoiXlat), idiosyncratic knowledge may enableUft immediate solution 
solely by recognition. If the exercises are tioo' advanced, then the ' ■ 
studeat_^subject'8 limited, competence with theAangiiage and ^Jro^raoj^ing 
. Byazea may prevent hia fiop using heuristics successfully. ' . 

K ' Another T^^^'d probleo*JLs vhen^^r^lative to* ifistruction in a 
prograaming language, to "'^Jitroduce instruction on general debugging 
heuristics and test^ for its effects. If the instruction on heuristics 

♦ 

and testing are too early, then students will not understand how to 

» ♦ 

appiy the heuristica and test exercises will be too'* difficult for 

heuristics to have an effect. If the instruction and testing are 

• * • . ■ * 

delayed too long, then ther^ will be significant differences between- * 

students' knowledge of the heuristics induced fros their prior , 

'experience. In addition, test exercises difficult enough to require uie 

of the heuristics <and not iserely perti^nt idiosyncratic experiential 

knowledge) will be so coaplex 4hat analysis of ^ubjects^' behavior will. 

be akde taore ^roublesoae. The appropriate tiae to introduce the 

heuristic instruction is wfaen the students have a ainiaally sufficient 

background that allows tfaea to undecstand and use the' heuristics, but 

not to- have realized them spontaneoifely. Discovering Jthe features that 

identify that point* iti tlae is the problea o£ co^irse. 

In our experlaerit, preb4ntation of ttie tutorial and testing of 

it effects were probably too late for the few general heuristics we 

wanted stMents to ledrn. The behavior 'of the «p-TUIORIAL group and the 

coaa«nt8 of the TUTORIAL gr^up indicate that aany of theSubjects had 

already inferred soae of the heuristics included In the tutorial fros 

their fifteen .hours of progr^aalng experience in BIP. For f tuden^s 



^ learning BASIC in BIP, presentation of the tutorial (or. other 
instruction on debugg&g) probably. shjDuld commence from 7 to 10 .hours 
iifto the course. At that point, mos^ students .have worlced with-all the 
major, constructs of BASIC and" are familiar^ with 'tfje facilit4.;?8 of the.' 
BIP system, but hav4 workec^^n only" a few programs complete e\ough for a 
general strategy to be useful. 

A most fundamental problem for stadi&s of the effects of " 
teaching general debugging heuristics' remains the analWis of 
problem-solving da'ta. In attempting t^aluate the>role and effects of 
general heuristics in debugging, one 'is in fact trying to characterize 
not just Che result of the problei^kolving process, but the process - 
itself. Itf analyzing the chronologies, for the test exercises in the* 
experiment, we found that simple tabulations of behaviors such as ' , 
listing or running a program are not reliable ^ic^tors o^ the strategy 
being a'pplied by the subject. Only by examining the -structure and - . 
cofatent of actions co^jrising larger episodes were,«e ,aj^*le to judge 
whether particular heuristics (were applied and their contribution to" 
ultimate solutions. The r^le of content, or semantic^^ the scoring • 
process Virtually precludes automated chronology ai^alysls. ^ 

In our experiment, the collection of "thinking aloud" protocols . 
from subjects as they worked test exer^ses might .have provi^fSTMata j" 
that would have increased the reliability with which chronologies' weL 
scored. However. thi9^ would have increased the already substantial cost 
of data analysis. For experiments witi sample aizes great enough to 
allowjstatistical evaluation of measures abstracted from chronologies, 
the cos.t of .collecting and examining thteklng-al'oud protocols would seem 
prohibitive. -Furthermore, for a large-s^ale i^dy' integrated into V 



1> { * , -? 

real-life instruo^Lonal system, the collection of th^ing-aloud 
' protocols wbuld dest^poy the advantage of inobtrTSiv^ess obtained 'by the 
•invisible" recording ,pf prograitoing chroffologi'es. 



Further sm^l-scale studiesjsU^ .t^at described here could ^ 
^ provide a relatively, fntormal an4 subjective^ evaluation of materials and 

methods for teaching debugging knowledge in an explicit manner. The 
^ main problems'^remaining to be solved are how to determine sensitive test 

matetisas and how to analyze complex problem-solving data 
c^tlprehensively and reMably. Although we were unsuccessful in our 
efforts, one goal that shoul^d be pursued is the development of process . 
<aodels for describing debugging behavior in specific domains, *Such 
models could be employed to represent changes in an individual's ^ 
behavior as the result of ln<h:ruction, and to contrast' ^Re behavior of 
individuals in different instruction^ treatments. 

As for a large-scale, formal statistical evaluation of whether 
teaching debugging 'directly is "worthwhile, there are additional, 
problems. Since th*e constraints of academic research make it difficult 
to gain access to a, large, representative Student sample, instructional 

T developments ;6hould probjBibly be evaluated outside the research 

c - 

enviroment. On^e an informally validated method for teaching debugging 
i^, Available, it sho'Uld be integrated into a real instructional program* 
^ * ^ecavise of the methodological and ethical difficulties of conducting * ' 
multi-group stifflie^ in an actual educational setting^ evaluatioa of ' ^ 
student perforpance would best be made relative to previous groups of 
students* Even if these problems^ can be overcome, the <fata an^ysis 

, ^ problem r^aains* It is unlikely that intensive methods suitable to 

• ^ - ^ * ' ' * * * - f 

small-scale studies (e*g*, prpcess models) will be feasiblcJIJor larga 

■ , • • * ' " ■ 
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experiments. This will limit the a^lyses in largT^udies to gross 
mea^res of learning, such as tot^l scores on in-class .exaainatiooft. 
Our judgeoent for the present ^s that the state-of-the-ari is still 
remote from a def*t(itive large-scale evaluation of how direct 
instruction in debugging, or other compl« problem-solving, will affect 
the abilities of studenta. * < 
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After you have vritten a pn>graa, you need to test it to make sure 
there ai^e errors, or ''bu^", in it* Hatty programs are designed 
to nm feore than once. For exai^le, som progra^^are written 
to cospute payrolls and mist be run at ^^end of ^ry pay period; 
other programs are written ^to.tabulate^udents' grades' and are run 
at the end of each grading period, 

% Since the coi^itlons .under which a progra is run ^11 not fce 
EXACTLY t^^e ^se each tlse the progras ife run, it is i^>ortant 
to realize that just because a progras works correctly for one 
set of conditions, you cannot assuse that it will work correctly 
under all qther comiitions* ^ .v 

For example, in sose programs- different kinds of input eai^e different 
^paxts *of the progras to be executed; thti to check a p^rogram yoFU ne^ 
to run it using all possible types of input for *^ch the prograi^ was 
designed • Ycm rust test every possible pathway throu^ the program. 

TEST THE PE0(3UH iflTH ALL POSSIBLE TtFES OP IHWm FOR SmiCH 
IT IS DESiaUED. * . 

* 

The following prograa desaoastrates how dif feceat inputs cause different 
parts &£ the- prograa to be executed. ' * • 7 

10 X -^IKT(gHD * lOOl) 

20 PRIST "I m THIHKISG 0? A KUMBES BETrfESS 0 ifiD 1000."' X 
30 L - Q . ' . y . 

40 e - 0 ' _ - 

>0 ?Rim "WHAT DO TOO THIHK H? HDKBE2 IS? " ' 

60 IHPOT G . ■ j# 

7q|,lF G - X TH£H*230 ■ --y ~ 

80 IF C "> X THES 160 , ' ^ 

90 IF-L - i TMS 140 ' ' 

100 P^T "TOO, LOW; OTESS % ' 

110 1 ' ' . ' ' • * , 

120 H - 0 . . * ' 

130 GOTO 60 . ' 

i&o ?ai8T ni's still too low. 'goess agait' 

.150 GOTO 60 * X 

IW IF H - 1 THEH 210 , X 

170-felMT "TOO HIGHi C0ESS AGAIB" • \ • 

180 H' - i ' • 

190 L - 0 » 

209 GOTO ^ . ■ . 

210 PglHT "YOO'SE STIU. TOO HIGH SO GUESS AGAIM" - " 
220 C»TO 60 

230 PRIST "sifflTi m nms^ is "jx 

240 ESD ■ • ^ 
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This prograa gBnerates a randoa integer (X) between 0 and lOOO^ 
The user then tries to ib^ss the number Uine'60, IHPUT S). If 
the user guesfses the nusber correctly (ll|ie 70), then line 230 
is executed, ^nd the prqgraa prints "RIGHT., Otherwise, if 
the user guesses a nu^er that is. too high, then line 160 is 
executed. If the preceding ^ess was also too high (which is the 
case if a - 1), then line 210 is executed, 'TfOU'RE STILL TOO HIGH 
SO CUESS AGAIN'l is prints, an* line 220 causes a jusp back to line 60, 
the, preceding gutss was not too high (if H is oot 1)^ then line 170 
is executed and 'TOO HIGH; GUESS ASAIH" is printed • AND SO OK. 

iSiecking the "(^JESS KY HUMBERT' prograa requires' that every possible ^ 
class of input be tested; i.e., an input' (gyess) that is lower than 
the qusber generated (X), another consecutive input that is still 
l^er than X, an input that is higher than X, another consecutive / 
input that is still nigher than X, and an input- that is equal to X, 

TEST THE PROGR^ WITH THE EXIREHE VALUES THAT THE IKPOT 

Initially, it is a good idea to test a program with the extrese 
values that the input can have. It is usually nor bard to think 
bf the extreme tjrpes of input -i^ich your program mist- handle, "and 
this test say reveal errors in vour program. In the ''GU^S MY NGHBER"* 
p^ogra^, the two extrese input values (guesses) are ''0" and "lOOO". 

If, during the testing of your progras with different inputs, the 
^output is ever wrong, then there is southing wrong in your program- 
'You miSt then try to characterize what is wrong.' ^ 



.( CHARACTERIZING THE ERROR 

CKmCTERIZE THE' WAY THE ERROR (S) SHOWS UP Is'tERKS 
-OF .THE IHFUT;aHD OUTPUt* 

Before jfou try to deterainr which part of the progran is wrking 
incorrectly (unless it's icsediately obvious), you should describe - 
what is vrong with the output, ^or exai^le, in the last progras, 
if you input a guess of^ 0 and the prograa prints **TOO HIGH", your 
d^criptioD would include the fact that the output is 'backwards for 
a too-low guess. If, in addition, the program said **TOQ LOW" in 
response to as input , of 1000, then you could characterize the erroneous 
behavior as being Vrorng for boph too-low ^nd too-high guesses* 
Describing. the "sy^tos" carefully is very helpful in leading you to 
locate its cause {the bug in th% prograa)? ^ process is siailar to a 
doctor asking questions about the exact location and na^ture of you%^paii 
before s/he be^ns to choose the appropriate treatjBcnt. ' 
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Siace the output, is the result -of following, the steps of the program^ 
if you can characterize how the output varies f ros what it should be, - 
given a particular input, then that may indicate which part of the 
prpgraza isn't doing what- it was intended to do. In o^der to 
.characterize the error(8) in a prograa, you should test it with 
different types of input iq order to see how different kinds of input 
affect the output. For exae^^le, perhaps the output is correct <§r 
closer to* the correct answer for certain inputs than it is for other 
inputs.- if so, then it is i^ortant to ask how the inputs that s^ive 
correct or "aore correct" answers differ froia the inputs that give 
•'less correct" answers.. If these* two inputs require different parts 
of the progran^ to be run, then tW*could guide you to the part of the 
progran that is npt workj^ng as it was intended. / 
• * 

sometimes" a program gives the CCmRECT OOTPOT FOR SOME INPUTS 
BUT NOT FOR OTHERS. WHEN THIS HAPPEHS YOU SHOULD EXAMINE THE 
^ DIFFERENCE (S) BETWEEH THE IKPUTS FOR WHigH THE PROGRAM WORKS 
, AND THE. ONES FOR WHICH IT FAILS* \ ' c 

The following progran was written to give chaqge ty a custoser -' 
when the iten being bought costs less than a dollar. The change can 
•be in half dollars, quarters, diiaes, nickels, and pennies. The prograa^' 
is designed to print the arsount of , chaoge in cepts and then give the 
fewest possible coins in change. ^ 

10 PRINT "TYPE THE PRICE OP YOOR ITEM. IT SBOULD- BE < 51,00" 
20 INPUT X 

30 LET C - 100 - X - . ' , 

40 PRINT "YOUR CHANGE FROM $1 IS " j C j "CENTS " 
50 LET H - 0 , , . 

60 LET Q - 0 ' 

70 LET D - 0 * 
80 LET N 0 ■ ' ' ' 

90 IF C< 50 THEN 120 ^ 
100 H - H+1 . 
110 C- C-50 ^ " 

120 IF e< 25 THEN 140 
130 Q-Q+1 1 , / 

140 IF C<10 ISEN 180 , 
• 150 D -Jfcfcl ■ , ^ ' ' 

160 C - C-iO . ' 

170 GOTO 140 * ' ^ 

180 IF C < 5 THEN..210 
190 H - N+1 , 
200 C - C-5 ' 9 % 
210 PRINT' "HERE IS YOUR CHANGE" 
220 PRINT H HALF DOLLARS" 
230 PSttNT Q QOAXTERS" 



240 P^INT D DIMES" 
250 PRINT N NICKELS' 

260 PRINT C I" PEKHIfiS' 
270 END • 



/ 



22 



The programmer might decide that a good first test for this program' 
,wouId be the case .in which one of each coin should be returned to the 
customer (1 half foliar, 1 quarter, 1 dime, 1 nickel, and 1 penny, for 
a total of 91 cents). So price of the item (the input number) imjst be 

Input; 9 " 
Output; YOUR CHAKGE FROM $1 IS 91. CEHTS > * 
, HERE IS YOUR CHAKGE / 

1 HALF DOLLARS X , 

1 QUARTERS , * 

4 DIMES ' ' 

0 NICKELS 

1 PENNIES r- . 

/ 

It is imnetiiately apparent that, the wrong number of dimes and nickels > 
has been recurnwi. inis might lead the prograooer test the program 
with atr input which should" return a dime- and a nickel. 

Input : 85 

Output: JSfOUR CHANGE FROM Si IS, 15 CENTS 
UERE^IS YOUR CHANGE 
0 HALF DOLLARS 

0 QUARTERS^ 

1 DL<1ES "x^ - . • 
1 KICKELS • - 

0 PENNIES 

rne output is correct, so the problem certainly isn't with the dimes 
and nickels alone. Before the program is run again, the first test, 
the one witn the incorrect out&ut. should be re-examined. Evidence 
about the nature of the e^ror^ighc have been overlooked because of 
Che obviqply wrongr number of nickels and diiaes in the output. Th-e/ 
programmer migTit add. up the coins to see how much change in cents was 
actually ret;»med iotjje first test and find the total to be 116 cents 
rather than 91 cen^g. The difference between tjhese two sums is 25 cents, 
and this might- suggest to the programmer that the error ds related to 
Che extra 25 ^cents. At this point the program sOould be examined for 
an er;-or related to the '25* cents' calculations." While .reading through 
that part of the program, the programmer should notice that Inother line 
is needed between 130 and 140 to subtract 25 frofe the total cents left 
at tft5g.poinc, or C. The absence of that line caused an extra 25 cents 
in thf output (since when a quarter was given in change, 25 cents was 
not subtracted from the total cents still owed the custoaer). After 
this change, the testing of thejjregram should be continued. 



AFTER A CHANGE, RETEST THE PROPRAM USING ALL POSSISLE 
TYPES OF INPUT FOR WHICH THE PROGRAM WAS DESIGNED. 
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After you've characterized the wrong output, located the section 
of code that you believe is responsible for the erroneous output, . 
and Qhanged that code to correct the error, the prograa oust be 
t^ed once again for all possible types of input. YoU aust retest 
your .program thoroughly for several rea8ons:_jEor exknple, you may ' 
have corrected the prograa so that it works for only one or tw 
additional types pf input; or the program a^y not work ^f or soae * 
inputs that were handled correctly beforfe your change, i.e., your - 
change interacts with a portion of the prograa that was executing 
correcciy before the change and now makes ^it give erroneous outnat. 
The prograa aust be retested wltt all types of input, even thosF 
that were handled correctly, before the change. 

The following program, which deaonstra'tes the iaportance of retcsting 
after a change, asks the user to type ik two auabers and tells hia/her 
how many numbers life between .the two numbers (inclusive). For eXasple 
there are 3 numbers between 5 and 7, i.e., 5, 6, and 7. 

10 PRIHT "TYPE TWaTlUMBERS, im \ WILL TELL YDU HOW iiAKY"^ 
20 PRINT "NUMBERS ARE BETWEEN YOUR TWO NUMBERS (INCLUSm)." 
30 INPUT X,Y ' N - ' - 

40 IF X < Y THEN 80 

50 H » X ' ' 

60 L = Y. • ■ 

• 70 GOTO no ' ^ ^ . • 

80 a »f Y 

;90 L - X : . 

100 P • L 

no N • 1. 

120 P - L + 1 _ 

130 N - N + 1 ' . . ' 

140 IF P < H THEN 120 

150 PRINT "THERE ARE ";«N; " NUMBERS BETWEEN "l L: " AMD fl > 
199 END , , ^ , n ^ 

The user types two nuabers, which are assigned to the variables 
X and Y. The variables' H and L are used to hold the high and 
low numbers, respectively. So, if X i%>high^ than Y, its 
value is assigned to H and the value of Y is assigned to Lj Lt 
Y is higher than X, the H'and L assignments are aade in" the 
opposite directiob. The variable P is used to count froa the low ' 
number up to the high riuaber,^and H is used lb keep track' of iiow 
many numbers are encountered along the way. Thus, if the user 
types 5 and 6 as the 'X and Y input, L becoaes 5, H becomes 6, P 
.counts, froa 5 to 6, and H ends up with %, , ^ - 

* ^ 

When the prograa is. run, it gfves the correct output only when th^- 
two ntiiaers adjacent to each other, e.g., "5" and "j^', or " 
. "6" ahd "5". The output is THERE ARE 2 NUMBERS BETWEEN" S-AHD 6". * 
j^Any^air of non^adjaceirt: nuabers causes an e^ror aessage to be printed, 
*^J^lch says that the 5»<4ra^ght be in an infinite loop. 'Die ^ 
cograaoer charatterlzes theN«:KsrT8 occurring when any two • 
jn-adjacent A.uBbers are given a& input. 

» ' _ * - 
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In the pro grant above, where only adjacent numbers X'^nd Y (bitth. X < Y 

and X > Y) give the correct output, the programmer might go through 

the following reasoning process while looking for the error: - 

-AHA, P doesn't get ^t to L when X > Y^ so line 70, should branch to 

lirfe LDO, ' . " ^ 

(The programmer changes line 70 to GOTO 100, and runs the program 

for X < Y and X > Y. S/He gets the same results as before, i^e., 

the program gives the correct output for adjacent* pair§ of numbers, 

otherwise it seems the program is in an ioxteite loop*) ; * ' 

-Well, same errpr, perhaps line lOO^^is^t^erf luous, since line 120 

assigns a value to P, so-^'ll deletetine 100, undo the previous change 

so that line 70 £s GOTO 110, and run the program again* 

(The result of testing the program is the same as before^ it works for 

adjacent number pairs, but every other pair'^iVes iafinite loop message.) 

-AHA,' line 120 should be P « P^+ 1, otherwist& P ^is always reset to 

equal L, the lowest number, plus 1, and P can^^ver reach H unless 

H is L+1! I'll change liner 120 and run the.^-program again* 

(The program gives the error message "Line 120 VARIABLE WITHOUT A KNOWN 

VAUm— P" for both X < r and^X > Y*) 

-Hmm. That's the first time I've g^i*tten that message. Why does P 
suddenly not have a value? I knov! P was L+l,>nd J ch^ged it to 
P»P+lj so the line thai I deleted, which set P equal to L, is necessary* 
I'll put line 100,. P »^L,.back ihto the program and run it* 
(^/He tries several pairfe of input, e*g*, 5 and 6, 5 and 8, 4 and 9j 
and they all work* ^Unfortunately, cases in which X > X aren't tested*) 
-Sutcess! It finally wdrks. ^ * 

The progr^ was fixed /for one type of inpup^ that is, for cases in 
Hhich X is Ifess than "Y; but two other types of itipiit were not tested^ 
X greater than Y and X. equal to Y- If exks^les of these two types 
of Input had been tested, the error message "Line 120* VARIABLE WITHOUT 
A KNOWN VALUE — P" would have told'the prograny>er that P still wasn't ' 
being assigned.^ Further examination of the program would have shown 
her/him that line 70 should, indeed, branch to line 100, so that P gets 
an initial value when X > Y and X » Y* Thus all types of input for 
w{iich a program is designed 6ust be retested after a change is Imade* 



Somettees you make a change to, the program, and the outj>ut is still 
wcong. You have to make Che choice between leaving the change in the 
program or returning the program to its state before the change* 

Take the program, for example, which t«lls the user how many numbers 
•are betwe^.n two input values. 
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10 PRINT "TYPE TWO NUMBERS, XND I WICL TELL YOU HOW MANY" 
20 PRINT "NUMBERS ARE BETWEEN YOUR TWO NUMBERS (INCLUSIVE)." 
30 INPUT X,Y 

40 IF X < Y THEN'SO , -U • 

50 H = X ^ \ ■ . ■ * ^ ^ 

60 L = Y • ' • " ; 

70 pOTO 110 " A , 

80 H - Y 

90 L - X ' ' 

100 P - L ' ' - ■ 

110 N - 1 ' , . 

i20 P«« P + 1 , 

130 N - N :f- 1 I* ... 

140 IF P < H THEN 120 *\ 
,150 PRINT "THERE ARE "; N; " NUMBERS BETWEEN Li " AND H 
199 END ' 

Suppose that a beginning programmer is told that this program has 
an error and is asked to find and correct it, S/be might notr-have 
'these guidelines for finding an errof • Since the program is short, 
s/he Bight decide to examine the code before .running the program. 
After doing this, the person might say "This ^uals business in lines 
30 through 90 is confusing. Seems to me theyVre double assigning 
things. H and L are being given two values.. L I think maybe 50 ^nd 60 
can be deleted. I'll try it." 

After dele ting, lin,es 50 and 60, the program is run. For Inputs 
where X ithe first input) is less than Y (the second input), the 
correct answer is giveh, and for all other inputs, the error message 
•'Line 120 VARIABLE WITHOUT A KNdWN VALUE—P." is printed. 

Since thd program has not been corrected by the,cl>ange, and even sore 
§^rors may bave been Introduced into the progra<^he change should * 
be uadone .and lines 50 and 60 restored to the program. 

Hie reason given for deleting lines 50 and 60, i.e^, th^t H and L 
are each bein^ given two valuefi, is true of cpurse, but the person did 
not examine the program carefully enough, because s/he did not notice 
that the values given to H and L in lines 50 and 60. are used in 
one pathw;^ through the program; and the values given in lines 80 
and 90 are used in a different pathwaTy through the program. Going 
through the step-by-step executiotf of a program (exactly as the- 
computer would) is a very valuable way to find errors* Howev^i*, 
after a superficial examination' of a program, deleting a line 
is probably a bad^idea* The person writing a program usually has a 
reason for putting in each line^ and before- you delete a line, yot^ 
should understatta the intended purpose of that line. . ' 

The programmet should have run this, prograa "before exaaining the code. 
The error message would have given ^er/bla the inforaatien tjiat P 
was not Being defined when either X > Y or X - Y. This Information 
points out which pa<:hway. through the prograa Sontalns the error. 
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THUS, EVEirir A' PROGRAM IS SHORT 'Ain) EASY^TO TRACE BY HAi^,'.". 
YOU SHOULD FIRST RUH.IHE PROGRAM, (ERROR MESSAGES.' AS 'WELLi; 
AS A CHARACTERIZATION OP THE ERROR IN TERMS OP INPOf AND 
OUTPUT, CAN BE VERY HELPFUL IN FINDING AN ERROR.) 

THEN 

IF YOU HAKE A CHANGE TO A PROGRAHT AND IT STILL GIVES THE 
SAME ERRONEOUS OUTEOT. REgTORE TfiE PROGRAM TO ITS STATE 
BEFORE THE CHANGE. YOU HAVEN'T POUND THE -ERRORCS) IH THE « 
PROGRAM, AND YOU'mAY HAVE INTRODUCED A NEW ERROR. . 



Sonetioes, when you' make a change to correct a program, the output 
'will still be wrong a£ter the change, but you should leave the change 
in the prograa, (Obviously, if you see any typographical errors that 
you made while typlrig in the prograa, you should correct those.) 

IF YOU MAKE A CHANGE TO A PROGRAM, AND THE OUTPUT IS 5TILL 
WRONG: IP THE CHANGE CORRECTS ONE PART OP THE PROGRAM (e.g., 
one pa'rt of the output), THEN LEAVE THE CHANGE IN THE PROGRAM 

It aay be the case that there Ib .sore ^han one error in tHe prograta, 
and you have found oce but not all of the errors/ Take the following 
progr'm as an exa^Le. W * 

10 PRINT' "THIS PROGRAM TALLIES THE VOTES OF 5 PEOPLE."' / 
20 PRINT *'TD VOTE YES, TYPE 1; TO VOTE NO, TYPE 0," 
30 Y - 0 
40 N » D 

50 FOR I Ji- 1 TO 5 

60 PRINT "VOTE NUMBER 

70 INPUT \ 

80 IF V -''l THEN 100 

90 N - N I * 
100 Y - Y + I ^ ' t ' 

aiO NEXT X 
120 IF Y <> N THEN 150 ' * 

130 PRINT "TIE M)TE" . . - ^ \ ' * 

140 GOTO 190 , . 

150 IF Y < N THEN ISO ^ _^ 
160 mNT "THE" NO VOTE WINS" 

170. GOTO 190 . ' . ' ^ 

180 PRINT ''THE YES VOTE WINS" ' 

190 END ' . ' / 

This prograa tallies the YES and NO votes of 5 people, then 
prints whether the 'YES 'a or 'HO's win. Tfte user inputs th*e 5 
votes. He types 1 for «'YES vote and 0 foiT a HO vote. - 
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Th,e program is run. When all the votes are eltlter YES' or N{f then^" 
"TIE, VOTE" la. printed. When the number of YES votes input is greater « 
than tl^e.number of NO votes, -"THE HO VOTE- WIHS" Is printed. When the • 
numbec of NO v<3«e% is 'greater than the nuoWr of YE^ votes, "THE NO 
VOTEWINS" is printed. This, program does the wrong thit* for three 
of eKe four different kindfe^of. input! 
d • ' — 

Since the program gives the correct out^Tut when the number of NO votes 
exceeds the number of YES votes, i.e., ."THE NO VOTE WINS" (except In- 
.-the extreme, case where all the votes are NO),' the programmer might 
check (;o see why line 180. "THE YES VOTE WINS", is not printed when 
it should bfe. tfS/He looks_ftt-±lne 180 aa^ the line, itself, looks all- 

.o^^ . through the program to find tte line that goes to 

line I^, which is line L5(J. ■ S/He sees an error^ In line 150 tf Y, 
wfaW-tallles the YES votes, is LESS THAN N, which tallies the NO votes^ 
then line 180 is executed, which 'prints "THE YES VOTE WINS". Line 150 
shduldsay "if Y is GREATER THAN N then execute line 180". This change 
is made to the program, and it is run. 

^Srv^S*^'^"''""^""' IfES vote is greater than the NO vote, 

"THE YES VOTE WINS" is ptrintedj but when the NO vote is greater than 
-the YES vote, /'TI^E YES VOTE WlliS" is printed. It seems like .the same 
wrong output as before the change, only" switched around!" (As before, 
when all^5 vote^ are either YES or NO, a;"TIE VOTE" is printed.)^ 

The prograraner must decid4 whether "to leave the change or not. i.e.. • 
150 IF Y:>.N then 180; 180,PRINT "THE YES VOTE WINS". Since Y tallies 

55«c^^L^!^^'^^' ^ ^ > then "THE YES VOTE 

WINS SHOULD be printed. The programmer decides to l^ve the change " " , 
and look for errors in other t)art^>of- the program. • * 

In line 150 (which now sa^ ""IF Y > N..y), if N is greater than Y, 
then line 160 is executed, which' prints "THE NO VOTE WINS",' sp that ' ' 
part, of the progr^ is coinrect. '• V , 

This program illtistrates Xhe^l^ortance of testing the- prc^raft -with 
the extreme values that ^he input can have, in this case, 5 YES voTes , 
^L^.^^^' Hheneyeir'the.inBut is aftl.YES votes or all HO votes, - 
TIE V0T8 Is printed Xline.l30). The programmer looks for tKe line* 
that must precede. the execution of line 130. If line 130 was executed,' 
then Y a^lf must have been equal in line 120. With an. odd number of " 
votes, .thTs isn't possible. Because Y and N are'botii initialized to S 
(lines ^0,and 40), .something must be wif^g with the counting procedure. 
The programmer exaMnes ' the FOR loop, ^r.e the votes are counted. S/He' 
notices that fr the vote is NO, bothirahd Y afe incremented! So, theHk 
should be a lipe 95 which says "GOtO 110". The change is made,' The* 
different possible types of'ltoput are ret^sted. Success. 
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SUMMARY^OF GUIDELINES 



TEST TtlE PRO(S^ WITH 
IT I.S DESIGNED 




TESTMIG TH§ PROGRAM 



PSSIBLE TYPES OF IHPiJT FOR WHICH 



TEST THE PROGRAM, WITH THE OCTREME - VALUES THAT THE 
INPUT CAN HAVE. • • • 



CHARACTERIZING THE ERROR 

. - - , ' ■ - 

CHAEACTERIZE THE WAY THE ERROR (S) SHOWS UP IN T^S OF THE 
INPUT AND OUTPUT. 

EVEN IF A PROGRAM IS SHORT AND EASY TO T'RACE BY HAND, YOU 
SHOULD FIRST .RUN THE PROGRAM. (ERROR MESSAGES, AS WELL AS 
A CHARACTERIZATION OF THE ERROR IN TERMS OF INPUT- AND OUTPUT 
CAN BE VERY HELPFUL IN FINDING AN" ERROR.) ) • ' * 



SOMETIMES A /ROGRAM GIVES THE CORRECT OUTP 
BUT NOT FOR OTHERS. WHEN THIS HAPPENS YOU 
DIFFERENCE (S) BETWEEN THE JiiPUTS- FOR WHICH 
AMD .THE ONES FOR WHICH IT FAILS. , 



..some inputs - 
ieT^xamine the 

'ROGRAM WORK& 



AFTER A CHANGE, RETEST THE PROGRAM. US ING^L POSSIBLE TYPES 
^ INPUT FOR WHICH THE PROGRAM WAS DESIGNED. ^ 

IF YOU MAKE A CHANGE TO A PROGRAM, AND IT STILL GIVES THE 
SAME ERRONEOUS. OUTPUT, RESTORE THE PROGRAM TO ITS STATE 
BEFORE THE CHANGE. YOU HAVEN'T. FOUND THE ERROR (S) IN THE 
PROGRAM, AND YOU MAY; HAVE INTRODUCED A 'NEW E^OR. 

IF YOU, MAKE A CHANGE TO A PROGRAM, AND THE OUTPUT IS STILL 
WRONG:. IF THE CIJANGE CORRECTS ONE PART OF THE PROGRAM .fe.R. , 
one part of the output), THEN LEAVE THE CHANQE IN THE PROGRAM. 
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* Appendix B> * Study Quiz to Accompany Tjutorial Text 

• - • 

OPEN BOOK QUIZ 

1) After writing a program why should you test it witl) all *the 
different types" of input that it vas designed to handle? 



2) Testing a program gives the f ollowi^^^^sults: 
Input: 0 (number of days) 

Expected Output^: 0 dollars and 0 cents f 

Output: 0 dollars and 0 centB^^ 

Input: 1 (number days) 

Expected Output: 0 dollars and 1 cent 

Output: 0 dollars .and 2 cent^ 

Input: 3 (number of days) 

Expected Output: 0 dollars and 7 cents ' ^ ^ ^ 

Output: 0 dollars and 14 cents 

Input: 10 (number of d^ys) 

Expected Output: 10 dollars and 23 cents 

Output: 20 dollars and 46 cents 

Characterize the error in this prograo* . ' • 



3) 



•(b) 
(c) 



1 



If a progr^ta gives the correct output for sone inputs but not 
for others, you should (a, b, or c) ' ' . 

(a) Scratch' it and start over. 

Hope that a user will only use inputs -for which the program 
gives the correct output. , • -• • 

Examine the difference (s) between the inputs for whlcfi.the program 
works and the ones for which it fails. T 



Why? 
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4) After making a chan|e'to a -pr9graB why should you REXEST ^he 
^program with all types* of Input for which it was designed? 



5) 



Why 



If you make a chang^to a program in order to correct It, and it 
still gives the^SAME erroneous output', you should (a or b) 
Le^ye the change in the program. 

Restore Che program to its state before tjie^ change, 

4 



6) if you -are told that a program has an error in "it, and you are 
asked, to find and correct that error, what, is t;he first thing you 
should do -after reading the^ogram description? (a, b,- or c) 
\{a) Qo' -through the step-by-ste^ execution of the program by hand 
(as the computer would) JLrf order-to find* the error* ^ 
(b) Run the prograuS witri'tbe different types of input for which it 
was designed in order to characterize the error, 
''(c) Read over the program,- delete any suspicious looking lin^s, and 
run the program. 



/ 
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Appendix C: Intr<^duetlon to the Tutorial Text 



IHTROnUCTIOH 



This Is is attempt tcx give you inforaatlbn that will help you to 
find the errors In a-coapu£«r program laore easily. ThlB Inforaatloo 
will be presented la the fors of jules ^Ich apply to certain * 
situations, rules-of-thuab that have been formulated froa the 
expedience of prograsaers who have spent aany hours la searching 
for errors, or "bugs". In conputer programs. 

Once you know there is an error in your program, the goal is to find 
it with a Eiiniaal aoount of tiee and effort. Since It is very 
hard to foroaiize ALL the knowledge about finding bugs that an 
^experienced progranser would have, the tules oresenced here will be 
general rules that provide the best way to about finding the error (s) 
in a coaputer progran oost of the tise. They provide a general fraaeyork 
and as you gain experience, you will be able 'to add exceptions to tb4se 
rules. If you foilow these rules*, the process of finding the error iay" 
seen to takp longer than could j however, it is caich acre likely that 
you will find the error or ALL of the errors in the progras, and that 
can save quite a bit of tine in the long run. As you gain experience 
the process of finding the .error tsay go faster. 

TKese rules lead to the desired result (which is finding the error with 
^ ai-niaal aooun't of tiae and effort) -nost of the tine. Everyone 
enploys this type of rule when trying to solve problems. When there ' 
is Qore than one possible course of action to reach a goal, a person 

weigh the positive and negative effects of each action under 
consideration before s/he makes a decision. For exa^le, suppose you 
are in a strange city, you need to get from vbere you are to a hotel in 
another part of the city, and a sap of the city is all the information 
you have to help you plan your route. In that situation (going from 
one place to another in a strange city), a general rule-of -thumb you 
might have is to stay on main streets. - If you have this rule, it is 
because of knowledge you have gained (e.g., from your -am expetience, 
or frosa talking to friends, etc.), for exa^le this knowledge cottid be: 

(1) street signs are more visible on sain streets 

(2) if you get lost, it is easier to ask directions oa a main street 

(3) naitv streets are safer, if that* section of town is unknown to you 

(4) a backstreet route may make crossing intersections sore difficult. 
Even if your general rule is to stay on main streets in a strange city, 
you luay choose not to follow the rule in certain instances. Perhaps 
the most i^ortant consideration is getting to place X as quickly as- 
possible, and you choose a backstreet reute because it is shorter and 
will allow you to miss the rush hour traffic on the main-streets. The 
circumstances qnder rfiich you make a decision will vary (e.g., finding 
the "best" route, where "best*-' seaas one that fulfills certaib 
requireaents such, as "requires least amount of tlBe"J, and general rale» 
will not always give the best solution to a particular problem. If jfou 
are a beginning programmer who is t tying ' to fisd the errors in your 
program, since you have no programming experience upon *rfjicb to 
formulate general nrf.es for finding the error, being given these general 



i 



rules should' s^ve you both ti»#and effort. After you have xaore 
prograj^Qg experience, you vill be able to^add exceptions to these 
f rules • 
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Appendix Test exercise CHAKG£R 

rais program was .written to give chaage to^a custorcr lAen the -Itea 
being bought <osts less than a dollar. Ibe change can be in half 
dollars, quarters, dioes, nickels, and pennies • Ibe prograa should 
pri^t both the aaount of change In cents and then the FEWEST possibl 
coins in change* 



5/31/77 11:12:06 
19 

10 PSIHI ''Ti?Z THE FEICE OF YOUE ITEH* . IT SBOfULD BE < ST* 
20 ?Kim " (FdE PRICE SWmJD BE 13 C£hTS,'*E»G. , 25, 49.) " 
30 IHFIT X 
40 LET C - iOO - X 

50 ?RL^*T "YOra CBAKGE FROH $1 IS " | C ; " CEHTS," 

60 DATA 50, 'HALF-^LLA^", 25, "QDABTERS", iO, *DIK^" 

70 DATA 5, \hiCK£LS", 1, "FEHBIES" - ' T 

80 ?Rm "HERS. IS YOUR CHA^E" 

90 H - 0 

100 SEAD A ' 

110 READ D$ . . 

120 IF A » i THEH 180 ^ 

130 IF C < A THEK 160 \ 

140 N « N + 1 

150 C « C - A 

160 PEIHI K; " ";DS 

170 GOTO 9^ 

180 ?Kim C; D$ * 

199 EKD 
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Appendix lest Exercise j)RILL 

' -\ . TASK DRTtL' 

We want you to vrlte a BASIC ^rogratt that presents sijeple 
arithaetic probless — your ovn cdsputer'-asslsted lost ruct ion • progras; The 
required prograa will be longer and sore cotg>lex than those you have 
previously cospleted tn BIF, but you prob^ly vorked with all the BASIC i 
statements you vill need* You will have at aost #1 an^ 1/2 hours to work on 
the task during a single sitting at the terminal. Do your b^t to coeplete 
a prograa that satisfies the specifications given below (use the DEMO to 
see a fancy aodel progras in operation), bur you will be paid even if you ' 
can't do so in the allotted tise, (Given the tise constraint, one possible 
approach is to design your prograa and then is^lesent it in successive 
stages, adding ix>re advance feature at each stage; however, you are free 
to tackle the problets-in any ssanner you prefer.) 

After i and i/2 hours (or sooner, if you are confident your progras 
worked correctly), we will exaslne your progras and try it out, if your 
progras isn't satisfactory, you will have at »o8t another 1/2 hour to fix 



it. 



\ ■ ■ 

Ti^ begin work, signon to BIP and^t3rpe the c^^sand TASK DHILL. BIF 
will ^t print the text of the problem, as ^ does norBaliy: in^ead refer 
to the specifications given below in these instruct ions ♦ During your work 
you can use any BIP coasands except the following: «)D£L, MD£?, teP^ DEHO 
^EACE, Use RUN to^^ out your progrM as sany times as you like. "Since 
you can't use mRE, you will have to be the judge of whether your prcjgraa 
satisfies the specifications before* you are ready to have us look at It. 
Yotf aa^ ttse ^ BIP eanoal . Run the DEMO as of tea "as you Xike, but do not 
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/ • 

ask for the MODEL; if ^ use the MODEL coaaand. we cannot fiajr j:ou for ^our " 

vork. REP does' not work for this task, but PLOW does. • There is paper for 

• i 

you to do any scratch work you want to: please nuaber any. sheets yoQ use 
and turn theytn at the end of the session. Since the prograa you will 
write will be too long to LIST on the tensinal screen ^t one tiae, we have 
set-up the teletypes in the roda to provide hardcopy of your prograa {you 
nay use the' LIST coEsaad, but the output wUl go off the top of the' 
screen- u^e the 'HOLD' key on the terainal to stop-and-start the output). 
To obtain hardcopy. SAVE your program in BIP as a file and .then, at the 
teletype, type (as requested) your student nuaber and naae* of the file you 

f 

SAVE'd. You say list your prograa on the teletype as aany tises as you 
like, 'and write on' the listings,, but we want you to turn in xhe listings at 
the end of the session. t 
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Prograa specif icatioo^/'far TASK DRILL j 



1) Hie user selects whetiier he wants to do addition or 
tract ion probleas* ' ^ 

2) The user selects v^ether he wants problems t|iat 
involve 1-digit integers (1-9) or 2-^igit .integers (10-99, 
not 1-9?). The integers used in each problea are randoaiy 
generated* ; ^ / 

3) /The user specifies how sany problcos he will work, 
with ^ainiauQ-of .1 and a saxiawa of 10 pz»bleas» ' - 

4) Subtraction probless taust always have ^n answer 
that is equal to or greater than 0 (no ne^tive answers)* 

5) The answer to each problea is checked' and 
appropriate feedbacif is printed. , Feedback on incorrect" 
answers includes the correct answer. ' • 

When the user finishes, the number of 'problems he 
specified, the prograo prints his score asr nu^er and 
percent correct. f 

7) Assune that the user of the progras is naive and 
csay type invalid responses to any qufestiom asked by the 
prograia. The progras should not •T)low up" in these cases. 
In general, it should also provide clear quest ions^ sad 
pxi&<f*-pui;put suicabii^ for naive users, ^fcry to write a 
progras you would waat to show off to another prograissser. 
\It does not have tt> have all the fancy features of the DEMO 
progras, but should satisfy the requiremenfcs list^ here» 
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Appendix F. Test Exercise ARITH-<1alC 
PROGRAM ARITU-CALC - 



This 'program is s.upposed to act as a 'calculator for siaple 
arithmetic expressions te.g,, 9*8. 43/5+11, l(r2*777) which have no 
parentheses to org^ize thea. It is intended to perform' the operations., . 
' an expression in a left to right order; fot example, 10+2*6 first adds ,10 

and 2 to get 12 and then multiplies 12 by 6 to get 72. IJote that this. is 
different from the way BASIC evaluates such expressions (BIP. manual 11.12)." 
The program is intended to handle o»ly "well-formed" input from the user 
and is expected to behav4, unpreiictably if the Input contaijns bad 
- characters. The follwing are exai^les of expressions for which the 
program is and is not expected to work. 

SHOULD WORK FOR ROT EJECTED TO WORK FOR 

4+5 (no spaces allowed) 
334/667-23+8*3 4+(5*3) (no parentheses) 

8/0 (gives error message) 4.5+13- (no decimals) 

^ ' 4A7tl3 (illegal character) 

The program is coj^lex. The a^ln difficulty is that^e expression 
input by the user is a string, and strings in BASIC (and parts of strings) ' 
cannot be multiplied, added, etc. The string must therefore b^ analyzed to. 
find the strings of digits it contains (l<e,, the numbers in the 
\ expression) and then these stifTngs of digits must be "translated"- Into 

numeric values that 'can be manipulated with arithmetic operations. Part of 
this work i» done by a Subroutine in the program. ' BIP dijin't give you any 
work with subroutines (BIP Manual 11.22), and ve doTt expect you to 
unde«Md the one in this program. The vay In which it is used is - 
cpl^Cned b] 



expl^ned by the REM statements in the program. The errorfs) la' this 



progract is (are) not in the subroutine or in the first lines of the program 
^faich set u£ an array of values used by the subroutine * The error(s) 
is (are) in the part of the prograa delimited by the REM statements ^ 
containing stars (asterisks). The prograa can be fixed with only minor 
Bodif ications (extensive re-writing is unnecessary)* 

^ ^To^get the program into y^ur prograa space, say GET ARITH-(^iLC 
^ after ypu signon to 5IP. You may RUN, LIST, and TRACE the program as you 
please, but do not use FLOW. Make any changes. you wish; ifv^ any point, 
you want^^o get the original program back, then just say GET ARITH-CALC , 
again. ^ J 
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Apfi^'adlx G\ Post-experimental Questlopaaire for TUTORIAL group 

QUESTIOHKAIRE 

NAKE: ^ ' ' - 

hi? KG.: . ^ . 

(1) Do you feel like the Batei;;Ul you read during the first session 

vas useful to you in the subsequent tasks? (Circle the appropriate 
nufiber.) 

0- * ' * * 

Hot Useful . - Extremely 

^1 ^ Useful 

1 2 3-4 5 



(2) As you were testing and debugging programs during the sessions, 
did you follow the guidelines presented in the material? 
(Circle the appropriate number*) 

Never Always 

'12 3 4 



Did you find it difficult to reme^er the giildellnerf? i 
(Yes or Uo) 



If, so, did you refer back to' the lesson? ' 



(3) Was any part of the lesson difficult to understand, or unclear, 
etc,? 



If so, which part(s)'? 



14^ 



40 



r 



) Do you have any suggestions (criticisms), i;n general^ regarding 
the manner bf presenta>ion of the guidelines? • 



) Would it have been better if the guidelines had been given 
to you before you finished Che BIP course?. Please explain 
your answer. ^ 



J Do' you tliiik it would be useful to ^iave BIP introduce this 
- material ^ part of the course? n 

j Would you like to make any further comments on the three 
sessions ygn just completed or on the BIP course itself? 



/ 



f 




